[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509968133 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003075] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013080] *** VALIDATE selinux *** [ 0.021450] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025781] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] *** VALIDATE tmpfs *** [ 0.029488] *** VALIDATE proc *** [ 0.030236] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035030] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.039313] debug: unmapping init [mem 0xffffffff96259000-0xffffffff96260fff] [ 0.041167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043021] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045010] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047013] ... max period: 00007fffffffffff [ 0.048012] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050292] rcu: Hierarchical SRCU implementation. [ 0.052525] smp: Bringing up secondary CPUs ... [ 0.053577] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057697] smp: Brought up 1 node, 4 CPUs [ 0.059010] smpboot: Max logical packages: 1 [ 0.060017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132643] node 0 deferred pages initialised in 70ms [ 0.136094] devtmpfs: initialized [ 0.137287] x86/mm: Memory block size: 128MB [ 0.140297] gcov: version magic: 0x41383552 [ 0.144297] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146311] pinctrl core: initialized pinctrl subsystem [ 0.147251] [ 0.147701] ************************************************************* [ 0.150034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152009] ** ** [ 0.154010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.158009] ** This means that this kernel is built to expose internal ** [ 0.159009] ** IOMMU data structures, which may compromise security on ** [ 0.161010] ** your system. ** [ 0.163011] ** ** [ 0.165012] ** If you see this message and you are not debugging the ** [ 0.167012] ** kernel, report this immediately to your vendor! ** [ 0.169011] ** ** [ 0.171010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ************************************************************* [ 0.175629] NET: Registered protocol family 16 [ 0.176469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184144] cpuidle: using governor menu [ 0.186009] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188527] PCI: Using configuration type 1 for base access [ 0.191120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200156] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205059] cryptd: max_cpu_qlen set to 1000 [ 0.207318] ACPI: Added _OSI(Module Device) [ 0.208012] ACPI: Added _OSI(Processor Device) [ 0.210012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215767] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221227] ACPI: Interpreter enabled [ 0.222048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223009] ACPI: Using IOAPIC for interrupt routing [ 0.225080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227334] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237728] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248335] acpiphp: Slot [2] registered [ 0.250091] acpiphp: Slot [5] registered [ 0.251097] acpiphp: Slot [6] registered [ 0.252076] acpiphp: Slot [7] registered [ 0.253070] acpiphp: Slot [8] registered [ 0.254071] acpiphp: Slot [9] registered [ 0.255076] acpiphp: Slot [10] registered [ 0.256098] acpiphp: Slot [3] registered [ 0.257066] acpiphp: Slot [4] registered [ 0.258076] acpiphp: Slot [11] registered [ 0.260070] acpiphp: Slot [12] registered [ 0.261072] acpiphp: Slot [13] registered [ 0.262096] acpiphp: Slot [14] registered [ 0.263089] acpiphp: Slot [15] registered [ 0.264070] acpiphp: Slot [16] registered [ 0.266083] acpiphp: Slot [17] registered [ 0.268074] acpiphp: Slot [18] registered [ 0.269097] acpiphp: Slot [19] registered [ 0.271092] acpiphp: Slot [20] registered [ 0.272104] acpiphp: Slot [21] registered [ 0.274077] acpiphp: Slot [22] registered [ 0.275092] acpiphp: Slot [23] registered [ 0.276072] acpiphp: Slot [24] registered [ 0.277077] acpiphp: Slot [25] registered [ 0.278068] acpiphp: Slot [26] registered [ 0.279069] acpiphp: Slot [27] registered [ 0.281093] acpiphp: Slot [28] registered [ 0.282071] acpiphp: Slot [29] registered [ 0.283087] acpiphp: Slot [30] registered [ 0.284070] acpiphp: Slot [31] registered [ 0.285040] PCI host bridge to bus 0000:00 [ 0.287018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.290017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.299190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301880] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305107] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313628] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.318503] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.320016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326517] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329283] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334502] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.353886] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.359013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364767] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414244] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455536] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.463014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.469020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.487018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.496053] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.501010] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.507014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.530767] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.536012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.540014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585648] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.597016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.625496] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627267] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628192] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.629218] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.630124] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.633084] iommu: Default domain type: Passthrough [ 0.636851] SCSI subsystem initialized [ 0.639150] ACPI: bus type USB registered [ 0.641152] usbcore: registered new interface driver usbfs [ 0.643067] usbcore: registered new interface driver hub [ 0.644066] usbcore: registered new device driver usb [ 0.646234] pps_core: LinuxPPS API ver. 1 registered [ 0.647010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.650063] PTP clock support registered [ 0.652037] EDAC MC: Ver: 3.0.0 [ 0.653515] PCI: Using ACPI for IRQ routing [ 0.654760] NetLabel: Initializing [ 0.655000] NetLabel: domain hash size = 128 [ 0.656013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.658082] NetLabel: unlabeled traffic allowed by default [ 0.660104] vgaarb: loaded [ 0.661263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.662011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.668000] clocksource: Switched to clocksource kvm-clock [ 0.779843] VFS: Disk quotas dquot_6.6.0 [ 0.781261] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.783429] *** VALIDATE ramfs *** [ 0.784439] *** VALIDATE hugetlbfs *** [ 0.785724] pnp: PnP ACPI init [ 0.787862] pnp: PnP ACPI: found 6 devices [ 0.805835] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.809228] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812115] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.816550] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.818676] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.821785] NET: Registered protocol family 2 [ 0.824303] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.828639] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.832583] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.837968] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.841333] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843896] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.847184] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850090] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852664] NET: Registered protocol family 1 [ 0.855482] RPC: Registered named UNIX socket transport module. [ 0.857743] RPC: Registered udp transport module. [ 0.859323] RPC: Registered tcp transport module. [ 0.861279] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.863797] NET: Registered protocol family 44 [ 0.865332] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866809] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868963] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871301] PCI: CLS 0 bytes, default 64 [ 0.872874] Unpacking initramfs... [ 2.302770] debug: unmapping init [mem 0xffff9c9bbcc54000-0xffff9c9bbffbffff] [ 2.306406] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.308093] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.310346] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.797609] Initialise system trusted keyrings [ 2.799454] Key type blacklist registered [ 2.800971] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.810340] zbud: loaded [ 2.813043] *** VALIDATE nfs *** [ 2.814275] *** VALIDATE nfs4 *** [ 2.815769] pstore: using deflate compression [ 2.818614] Platform Keyring initialized [ 2.907615] NET: Registered protocol family 38 [ 2.909439] Key type asymmetric registered [ 2.910675] Asymmetric key parser 'x509' registered [ 2.912294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.915211] io scheduler mq-deadline registered [ 2.916691] io scheduler kyber registered [ 2.918237] io scheduler bfq registered [ 2.919924] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.922332] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.924679] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.926965] ACPI: Power Button [PWRF] [ 2.931625] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.938454] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.949383] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.956401] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.972000] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.000557] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.029781] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.033696] Non-volatile memory driver v1.3 [ 3.035048] Linux agpgart interface v0.103 [ 3.062638] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 3.065059] vda: detected capacity change from 0 to 74149888 [ 3.078192] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.080422] vdb: detected capacity change from 0 to 1073741824 [ 3.097190] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.099482] vdc: detected capacity change from 0 to 2621440000 [ 3.114159] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116945] vdd: detected capacity change from 0 to 2621440000 [ 3.129027] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.131036] vde: detected capacity change from 0 to 4294967296 [ 3.143124] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.145921] vdf: detected capacity change from 0 to 4294967296 [ 3.151195] libphy: Fixed MDIO Bus: probed [ 3.165957] usbcore: registered new interface driver usbserial_generic [ 3.168067] usbserial: USB Serial support registered for generic [ 3.169938] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.174226] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.175707] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.177636] mousedev: PS/2 mouse device common for all mice [ 3.180099] rtc_cmos 00:05: RTC can wake from S4 [ 3.183926] rtc_cmos 00:05: registered as rtc0 [ 3.185688] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.188052] intel_pstate: CPU model not supported [ 3.189732] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.192972] hid: raw HID events driver (C) Jiri Kosina [ 3.195167] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.196478] usbcore: registered new interface driver usbhid [ 3.200460] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.201584] usbhid: USB HID core driver [ 3.201746] drop_monitor: Initializing network drop monitor service [ 3.208494] Initializing XFRM netlink socket [ 3.211033] NET: Registered protocol family 10 [ 3.214560] Segment Routing with IPv6 [ 3.215760] NET: Registered protocol family 17 [ 3.217319] mpls_gso: MPLS GSO support [ 3.222861] RAS: Correctable Errors collector initialized. [ 3.224509] AVX version of gcm_enc/dec engaged. [ 3.225742] AES CTR mode by8 optimization enabled [ 3.302970] sched_clock: Marking stable (3302947857, 0)->(4218606269, -915658412) [ 3.305849] registered taskstats version 1 [ 3.307409] Loading compiled-in X.509 certificates [ 3.308909] zswap: loaded using pool lzo/zbud [ 3.332355] Key type big_key registered [ 3.343190] Key type encrypted registered [ 3.344919] ima: No TPM chip found, activating TPM-bypass! [ 3.346498] ima: Allocated hash algorithm: sha1 [ 3.347995] ima: No architecture policies found [ 3.349627] evm: Initialising EVM extended attributes: [ 3.351170] evm: security.selinux [ 3.352185] evm: security.ima [ 3.353173] evm: security.capability [ 3.354324] evm: HMAC attrs: 0x1 [ 3.356616] rtc_cmos 00:05: setting system clock to 2026-06-28 01:15:27 UTC (1782609327) [ 3.362625] debug: unmapping init [mem 0xffffffff97203000-0xffffffff973fffff] [ 3.365176] debug: unmapping init [mem 0xffffffff95f82000-0xffffffff96258fff] [ 3.371094] Write protecting the kernel read-only data: 28672k [ 3.374318] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 3.376544] debug: unmapping init [mem 0xffffffff94f14000-0xffffffff94ffffff] [ 3.406627] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.413502] systemd[1]: Detected virtualization kvm. [ 3.415188] systemd[1]: Detected architecture x86-64. [ 3.416807] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.441705] systemd[1]: No hostname configured. [ 3.443070] systemd[1]: Set hostname to . [ 3.444722] random: systemd: uninitialized urandom read (16 bytes read) [ 3.446587] systemd[1]: Initializing machine ID from random generator. [ 3.576988] random: systemd: uninitialized urandom read (16 bytes read) [ 3.579116] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.583444] random: systemd: uninitialized urandom read (16 bytes read) [ 3.585686] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.589649] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.178479] device-mapper: uevent: version 1.0.3 [ 4.181126] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.518691] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.995509] virtio_net virtio0 ens2: renamed from eth0 [ 5.132879] scsi host0: ata_piix [ 5.161276] scsi host1: ata_piix [ 5.162904] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.165071] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.053560] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.860636] random: crng init done [ 9.863534] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.392763] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.590379] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.817793] SELinux: Disabled at runtime. [ 11.877496] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.887077] systemd[1]: Detected virtualization kvm. [ 11.889048] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.341135] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.344563] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.349496] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.355335] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.359190] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.366617] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.370904] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ 12.473491] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.799217] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.077821] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.178570] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.253103] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.276376] EDAC sbridge: Ver: 1.1.2 [ 14.705294] Key type dns_resolver registered [ 15.005896] NFS: Registering the id_resolver key type [ 15.008304] Key type id_resolver registered [ 15.010078] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg139-server login: [ 39.822283] libcfs: loading out-of-tree module taints kernel. [ 39.843147] Key type ._llcrypt registered [ 39.844590] Key type .llcrypt registered [ 39.901897] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 48.277700] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 49.040437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.048932] alg: No test for adler32 (adler32-zlib) [ 50.827715] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 51.798633] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 53.569774] Key type lgssc registered [ 55.393333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.301691] hrtimer: interrupt took 4614954 ns [ 72.857475] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.749034] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 132.338392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.396050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.667335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.731593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.864436] Lustre: lustre-MDT0000: new disk, initializing [ 133.955324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.994354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 138.192613] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.877740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.058842] Lustre: 6485:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.137688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.145262] Lustre: Skipped 1 previous similar message [ 152.273623] Lustre: lustre-MDT0001: new disk, initializing [ 152.335861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.378975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.396546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.947025] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.386456] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.601812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 173.980295] Lustre: lustre-OST0000: new disk, initializing [ 173.982858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 173.996438] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.014967] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 174.080228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.190522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.197457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.258043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 181.119948] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.442094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.603803] Lustre: lustre-OST0001: new disk, initializing [ 196.610547] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.617747] Lustre: 9499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.697550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.860513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.875163] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.961185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 204.818569] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.134417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.262071] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.417811] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing check_logdir /tmp/testlogs/ [ 236.012882] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing yml_node [ 241.085585] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 243.681326] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 246.623899] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 248.243778] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jun 27 21:19:31 EDT 2026 [ 265.902101] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 267.446220] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.664401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.677334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.696703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.181773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.210779] Lustre: Skipped 2 previous similar messages [ 290.153062] Lustre: server umount lustre-MDT0000 complete [ 294.196704] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609618 with bad export cookie 11333224961161223698 [ 294.198122] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 294.217069] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.415594] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.417351] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.433968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.433978] Lustre: Skipped 4 previous similar messages [ 296.467070] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 299.427300] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.427300] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.461462] Lustre: Skipped 2 previous similar messages [ 300.849696] Lustre: server umount lustre-MDT0001 complete [ 305.083307] Lustre: server umount lustre-OST0000 complete [ 308.990388] Lustre: server umount lustre-OST0001 complete [ 315.643476] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 326.265878] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 368.562402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.030213] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 369.066663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 369.147235] Lustre: lustre-MDT0000: new disk, initializing [ 369.247471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.260583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 373.822701] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 384.092949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.258501] Lustre: 15051:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 384.323120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 384.329352] Lustre: Skipped 1 previous similar message [ 384.444392] Lustre: lustre-MDT0001: new disk, initializing [ 384.588598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 384.636714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 384.651796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 388.805432] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 400.213924] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 407.961439] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 415.359741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 415.651945] Lustre: lustre-OST0000: new disk, initializing [ 415.655651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.665109] Lustre: 17020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.751817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.815400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.826626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 417.058795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 423.240231] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 433.695473] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 438.742103] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.980507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.769596] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.935300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 447.973089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 447.973135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.978149] LustreError: Skipped 1 previous similar message [ 447.987334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.987565] Lustre: Skipped 2 previous similar messages [ 458.212458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.220657] Lustre: Skipped 3 previous similar messages [ 460.255087] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 460.468126] Lustre: server umount lustre-OST0000 complete [ 466.401051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.409938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.576442] Lustre: server umount lustre-MDT0000 complete [ 473.574937] LustreError: 18360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.595748] LustreError: 18360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 474.777198] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609798 with bad export cookie 11333224961161225231 [ 474.787855] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.799837] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.179747] Lustre: server umount lustre-MDT0001 complete [ 480.290740] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:23:23 (1782609803) [ 483.218521] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 485.502881] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:23:28 (1782609808) [ 491.872095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.190527] LustreError: 19433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.263884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.723570] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 497.634567] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.704453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.015689] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 524.188357] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 530.704192] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.457674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 544.981477] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 552.771862] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 572.384700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 572.385668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 572.393901] LustreError: Skipped 1 previous similar message [ 572.395633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 572.404532] Lustre: Skipped 2 previous similar messages [ 572.420478] Lustre: Skipped 5 previous similar messages [ 572.673908] Lustre: server umount lustre-OST0000 complete [ 582.632529] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.648025] LustreError: 19434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 584.697963] Lustre: server umount lustre-MDT0000 complete [ 589.163263] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782609913 with bad export cookie 11333224961161226113 [ 589.181724] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.712793] Lustre: server umount lustre-MDT0001 complete [ 603.840156] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 607.397862] Key type lgssc unregistered [ 607.622719] LNet: 22989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.628647] LNetError: 22989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.683783] LNet: Removed LNI 192.168.201.139@tcp [ 609.994472] Key type .llcrypt unregistered [ 609.996289] Key type ._llcrypt unregistered [ 636.203780] Key type ._llcrypt registered [ 636.205495] Key type .llcrypt registered [ 636.318244] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 637.549189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.721987] alg: No test for adler32 (adler32-zlib) [ 638.821589] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 639.055306] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 640.751230] Key type lgssc registered [ 641.808867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.504972] loop: module loaded [ 649.586249] loop0: detected capacity change from 0 to 204800000 [ 649.800912] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.807788] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 650.169281] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.204316] loop0: detected capacity change from 0 to 204800000 [ 656.457679] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.465832] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.472957] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.961845] loop0: detected capacity change from 0 to 204800000 [ 667.431496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 667.453810] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.683582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 668.735691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 668.836742] Lustre: lustre-MDT0000: new disk, initializing [ 668.930715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.951181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 673.632699] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 683.633587] loop1: detected capacity change from 0 to 204800000 [ 690.302846] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 690.793493] Lustre: lustre-OST07c6: new disk, initializing [ 690.798142] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 690.804594] Lustre: Skipped 1 previous similar message [ 690.810566] Lustre: 26182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 690.943600] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 699.143405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 699.154606] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 699.196044] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 699.446814] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 704.713945] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 704.733481] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 704.742801] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 704.753500] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 704.759777] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 704.767402] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 722.223214] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 722.229742] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.238438] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.244666] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 722.252303] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 722.262593] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.870022] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 727.885211] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.896484] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.910684] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.925300] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.938063] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.957580] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 727.978619] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.995627] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 728.004597] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.026722] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.035279] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.479689] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 733.500862] Lustre: 25165:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 733.509950] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 733.519352] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.528826] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 733.536895] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.546816] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 733.556834] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.566185] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 733.587153] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 733.593987] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 733.610273] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.037453] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 742.051830] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 742.060801] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.069025] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.082797] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.088115] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.093140] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 742.098320] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.103326] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.107803] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 742.111845] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.115851] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.359355] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 750.370507] Lustre: 25164:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 750.377746] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.393187] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.414454] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.433561] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.448467] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 750.463977] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.482610] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 750.491103] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 750.497980] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.509135] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 806.893595] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.908464] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 806.915312] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 808.363810] Lustre: server umount lustre-OST07c6 complete [ 821.576299] Lustre: server umount lustre-MDT0000 complete [ 843.249439] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 846.420769] Key type lgssc unregistered [ 846.719833] LNet: 28740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.729542] LNetError: 28740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.746375] LNet: Removed LNI 192.168.201.139@tcp [ 847.680388] Key type .llcrypt unregistered [ 847.683501] Key type ._llcrypt unregistered [ 870.021871] Key type ._llcrypt registered [ 870.026387] Key type .llcrypt registered [ 870.146134] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 871.459504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.475552] alg: No test for adler32 (adler32-zlib) [ 872.580504] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 872.878462] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 874.599167] Key type lgssc registered [ 875.966137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 884.702027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 884.764535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.467864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.071595] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 895.969159] LustreError: 30016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.984890] LustreError: 30016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 901.110212] LustreError: 30017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 902.326521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.860364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 908.295273] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 918.072738] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.685415] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 933.977403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.463917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.111601] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 950.461792] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 955.953823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 976.354047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.357517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.379950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 981.245711] Lustre: server umount lustre-OST0000 complete [ 986.447277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.477468] Lustre: Skipped 1 previous similar message [ 986.509916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.523873] Lustre: Skipped 2 previous similar messages [ 991.712117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.720997] Lustre: Skipped 1 previous similar message [ 992.906898] Lustre: server umount lustre-MDT0000 complete [ 996.842076] LustreError: 30746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.865637] LustreError: 30746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 997.407891] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610321 with bad export cookie 7888557087447038513 [ 997.418828] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 997.795439] Lustre: server umount lustre-MDT0001 complete [ 1012.178826] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1015.931857] Key type lgssc unregistered [ 1016.430616] LNet: 33774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.440843] LNetError: 33774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.458965] LNet: Removed LNI 192.168.201.139@tcp [ 1017.268496] Key type .llcrypt unregistered [ 1017.271170] Key type ._llcrypt unregistered [ 1041.634552] Key type ._llcrypt registered [ 1041.639670] Key type .llcrypt registered [ 1041.787609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1043.246782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1043.659083] alg: No test for adler32 (adler32-zlib) [ 1044.790332] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1045.078516] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1046.775159] Key type lgssc registered [ 1048.052650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1056.508193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1056.520897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.897440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.867060] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1068.513335] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.542162] LustreError: 35044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1072.205253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.589477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.909638] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1086.192346] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1092.969996] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.185195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.695959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.766613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1109.760242] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1118.494630] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1130.465950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1130.475878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.498695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.512472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.521195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.525047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.553540] Lustre: Skipped 1 previous similar message [ 1134.467250] Lustre: server umount lustre-OST0000 complete [ 1139.168354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1139.178170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.193555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.754118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.755721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.794329] Lustre: Skipped 1 previous similar message [ 1145.128792] Lustre: server umount lustre-MDT0000 complete [ 1147.876524] LustreError: 35039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.907555] LustreError: 35039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1149.215942] LustreError: 35779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610473 with bad export cookie 643004558848609793 [ 1149.232415] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1149.235787] LustreError: 35779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1149.629381] Lustre: server umount lustre-MDT0001 complete [ 1162.293692] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1165.410426] Key type lgssc unregistered [ 1165.798554] LNet: 38579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.815985] LNetError: 38579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.833787] LNet: Removed LNI 192.168.201.139@tcp [ 1166.778824] Key type .llcrypt unregistered [ 1166.781596] Key type ._llcrypt unregistered [ 1193.487271] Key type ._llcrypt registered [ 1193.489756] Key type .llcrypt registered [ 1193.611724] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1194.858199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.077188] alg: No test for adler32 (adler32-zlib) [ 1196.092975] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1196.264367] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1197.919175] Key type lgssc registered [ 1198.950379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.625538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.649646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.189064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.554287] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1219.561543] LustreError: 39999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.589471] LustreError: 39999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1222.793180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.358174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1227.871683] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1236.926263] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1243.824295] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1252.437873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1252.820905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.883169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1259.461630] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1269.370715] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1275.743676] Lustre: server umount lustre-MDT0000 complete [ 1276.473857] LustreError: 42140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.436469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.456650] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.467148] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1279.971788] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610604 with bad export cookie 12225409645446866867 [ 1279.997380] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1281.544897] LustreError: 39993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.569676] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 1283.562682] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.588930] Lustre: Skipped 2 previous similar messages [ 1283.597330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1286.673374] LustreError: 39993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.674117] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [ 1286.697570] LustreError: 39993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1286.713493] Lustre: Skipped 1 previous similar message [ 1289.696965] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1294.304581] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1294.549731] Lustre: server umount lustre-MDT0001 complete [ 1315.827268] Lustre: server umount lustre-OST0000 complete [ 1335.712176] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1338.421795] Key type lgssc unregistered [ 1338.747731] LNet: 43858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.751382] LNetError: 43858:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.766846] LNet: Removed LNI 192.168.201.139@tcp [ 1339.753085] Key type .llcrypt unregistered [ 1339.755469] Key type ._llcrypt unregistered [ 1365.485773] Key type ._llcrypt registered [ 1365.487837] Key type .llcrypt registered [ 1365.566058] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1366.567539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1366.623827] alg: No test for adler32 (adler32-zlib) [ 1367.657663] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1367.908481] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1369.551282] Key type lgssc registered [ 1370.802791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.562823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1379.596885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.144797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.654818] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1391.589806] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.602847] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1395.717292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.222699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.310921] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1409.730878] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.724738] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1423.965054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.297900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.374146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1430.885512] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1441.142279] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1448.494271] Lustre: lustre-OST0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1452.002622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1452.003549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.013040] LustreError: Skipped 1 previous similar message [ 1452.015856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.063214] Lustre: Skipped 1 previous similar message [ 1453.574921] Lustre: lustre-OST0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1453.582233] Lustre: Skipped 1 previous similar message [ 1457.122428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1457.133209] Lustre: Skipped 1 previous similar message [ 1462.245461] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1462.246927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.269724] Lustre: Skipped 2 previous similar messages [ 1462.467611] Lustre: server umount lustre-OST0000 complete [ 1482.727853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.744980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.748228] Lustre: Skipped 1 previous similar message [ 1482.774718] Lustre: Skipped 1 previous similar message [ 1487.101124] Lustre: server umount lustre-MDT0000 complete [ 1487.843682] LustreError: 45271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.873119] LustreError: 45271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1490.932800] LustreError: 45257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782610815 with bad export cookie 13364671368366546046 [ 1490.942904] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.948222] LustreError: 45257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1505.247970] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1505.667406] Lustre: server umount lustre-MDT0001 complete [ 1519.474508] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1522.603368] Key type lgssc unregistered [ 1522.953769] LNet: 48990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.969409] LNetError: 48990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.983477] LNet: Removed LNI 192.168.201.139@tcp [ 1523.931466] Key type .llcrypt unregistered [ 1523.935951] Key type ._llcrypt unregistered [ 1554.893663] Key type ._llcrypt registered [ 1554.901657] Key type .llcrypt registered [ 1555.075513] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1556.735746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.909068] alg: No test for adler32 (adler32-zlib) [ 1558.133977] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1558.519628] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1560.359800] Key type lgssc registered [ 1561.399870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1569.937103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1569.958933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.744817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.466831] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1581.034453] LustreError: 50407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.057104] LustreError: 50407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1586.144997] LustreError: 50408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.285232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.692385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.001319] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1603.022813] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1610.711662] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1619.216661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1619.571301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.846599] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1636.300094] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1641.076829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1646.432887] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1650.848381] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1663.517311] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-2721h43m05s [ 1665.584361] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1669.600229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1669.608792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.630658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.695878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.706828] Lustre: Skipped 4 previous similar messages [ 1672.765497] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 1674.213866] Lustre: server umount lustre-MDT0000 complete [ 1676.781781] LustreError: 50407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.808750] LustreError: 50407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1678.386878] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1679.902707] LustreError: 51650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.018811] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.050045] LustreError: 50402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1686.681340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.888432] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.400914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.531376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1688.426017] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1692.294669] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1695.945512] Lustre: DEBUG MARKER: Wait for df (36606) ... [ 1697.943896] Lustre: DEBUG MARKER: done [ 1703.908791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.926338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1703.941251] Lustre: Skipped 3 previous similar messages [ 1703.956243] Lustre: Skipped 1 previous similar message [ 1709.031376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1709.039835] Lustre: Skipped 2 previous similar messages [ 1709.702955] Lustre: server umount lustre-OST0000 complete [ 1714.146950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.153041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1720.124700] Lustre: server umount lustre-MDT0000 complete [ 1720.289969] LustreError: 52899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.311238] LustreError: 52899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1723.936244] LustreError: 53036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611048 with bad export cookie 17333430880498903088 [ 1723.938981] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1723.957140] LustreError: 53036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1724.421758] Lustre: server umount lustre-MDT0001 complete [ 1734.648560] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1737.764993] Key type lgssc unregistered [ 1738.070594] LNet: 55345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.076726] LNetError: 55345:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.089657] LNet: Removed LNI 192.168.201.139@tcp [ 1738.846381] Key type .llcrypt unregistered [ 1738.852098] Key type ._llcrypt unregistered [ 1783.412609] Key type ._llcrypt registered [ 1783.417548] Key type .llcrypt registered [ 1783.602888] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 1785.237978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1785.423262] alg: No test for adler32 (adler32-zlib) [ 1786.760770] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1786.971129] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 1788.625107] Key type lgssc registered [ 1790.182571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1802.115428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1802.162669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.582869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1803.606503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1803.823281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.452863] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1818.161320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.317970] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1818.367260] Lustre: 57569:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.740614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.755973] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1835.502757] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1843.231150] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1851.808762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1852.143597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1852.506654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.177232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1860.036464] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 1869.251383] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1876.503350] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-2721h46m39s [ 1878.304982] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1887.011473] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1889.823701] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1889.890272] LustreError: 57578:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1910.344494] Lustre: lustre-MDT0000: Client 227321d5-901c-4f7d-921b-3fc270cfed12 (at 192.168.201.39@tcp) reconnecting [ 1929.919430] LustreError: 57578:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1932.730886] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1935.331869] Lustre: DEBUG MARKER: done [ 1946.084223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1946.097129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.135489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.598563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.877793] Lustre: server umount lustre-OST0000 complete [ 1956.832647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.835495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.852404] Lustre: Skipped 1 previous similar message [ 1956.863858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.873961] Lustre: Skipped 2 previous similar messages [ 1957.921037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.934867] Lustre: Skipped 1 previous similar message [ 1959.904889] Lustre: server umount lustre-MDT0000 complete [ 1964.019849] LustreError: 57562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611288 with bad export cookie 17026283674382427682 [ 1964.035116] LustreError: 57562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1964.037178] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1964.331161] Lustre: server umount lustre-MDT0001 complete [ 1979.655037] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 1983.295058] Key type lgssc unregistered [ 1983.679786] LNet: 61902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.692300] LNetError: 61902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.728888] LNet: Removed LNI 192.168.201.139@tcp [ 1984.828811] Key type .llcrypt unregistered [ 1984.831657] Key type ._llcrypt unregistered [ 2044.696558] Key type ._llcrypt registered [ 2044.703532] Key type .llcrypt registered [ 2044.823972] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2046.339588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.378188] alg: No test for adler32 (adler32-zlib) [ 2047.459909] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2047.744128] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2049.495283] Key type lgssc registered [ 2050.891789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2055.061929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2055.099525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.513784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2056.559651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2056.754364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.163714] Lustre: Failing over lustre-MDT0000 [ 2058.549746] Lustre: server umount lustre-MDT0000 complete [ 2066.780586] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:49:49 (1782611389) [ 2074.504535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.869445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.413799] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2090.267968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.449216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2090.494993] Lustre: 64708:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2090.829429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.746942] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2103.932915] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.545935] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2119.713295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.939056] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2120.116615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.285563] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2128.087252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 2136.067455] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2143.112621] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2152.416668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2152.421131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.430060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.956472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2153.958794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.964457] Lustre: Skipped 1 previous similar message [ 2153.979393] Lustre: Skipped 1 previous similar message [ 2156.833396] Lustre: server umount lustre-MDT0000 complete [ 2160.750161] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611484 with bad export cookie 17939296629747284647 [ 2160.751564] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.769911] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2161.261346] Lustre: server umount lustre-MDT0001 complete [ 2165.512770] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2170.369070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2179.096419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.575331] Lustre: 63629:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611488/real 1782611488] req@000000001984489d x1869203501001216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2180.595821] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.663520] Lustre: 63629:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611489/real 1782611489] req@000000007c8af2d8 x1869203501001344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611505 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.696334] Lustre: 63631:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611493/real 1782611493] req@00000000bc6e1c23 x1869203501001472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.766570] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf8f52f3d0073038a [ 2185.780408] Lustre: MGC192.168.201.139@tcp: Connection restored to 0@lo (at 0@lo) [ 2186.200737] LustreError: 66556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.217331] LustreError: 66556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2186.304602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2186.389995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2189.346756] LustreError: 66557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.394727] LustreError: 66556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2190.421541] LustreError: 66556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2190.636441] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2191.382099] Lustre: 63630:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611499/real 1782611499] req@00000000a8139258 x1869203501001856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611515 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.403287] Lustre: 63630:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2195.431892] LustreError: 66557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.450506] LustreError: 66557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2196.466122] Lustre: 63630:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611504/real 1782611504] req@000000008e3fc3ba x1869203501001984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611520 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.885284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.304196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2205.525927] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2205.681751] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2209.830123] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2215.157157] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.935946] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2235.362571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.368626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2235.388160] Lustre: Skipped 1 previous similar message [ 2235.408188] Lustre: Skipped 3 previous similar messages [ 2237.743510] Lustre: server umount lustre-MDT0000 complete [ 2240.488174] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2240.517419] LustreError: 68482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2241.650048] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611565 with bad export cookie 17939296629747286922 [ 2241.655550] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.669761] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2241.949045] Lustre: server umount lustre-MDT0001 complete [ 2248.443613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2258.795391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.959165] Lustre: 63631:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782611569/real 1782611569] req@000000000441f4f1 x1869203501031424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782611585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2261.008118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.048916] Lustre: Skipped 1 previous similar message [ 2266.489454] LustreError: 66557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.555914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.599734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2271.188609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2280.485339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.068179] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2286.690732] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2290.220618] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2295.942143] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2302.728956] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2311.650638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.662104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.675250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.751771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.757587] Lustre: Skipped 2 previous similar messages [ 2317.059364] Lustre: server umount lustre-MDT0000 complete [ 2320.869675] LustreError: 71164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.896250] LustreError: 71164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2321.996309] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611646 with bad export cookie 17939296629747288301 [ 2322.002897] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.010528] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2322.554615] Lustre: server umount lustre-MDT0001 complete [ 2330.790074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2335.611378] Lustre: DEBUG MARKER: files compared the same [ 2349.571490] Lustre: server umount lustre-OST0000 complete [ 2369.798799] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2372.579851] Key type lgssc unregistered [ 2372.854991] LNet: 74442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.860223] LNetError: 74442:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2372.875479] LNet: Removed LNI 192.168.201.139@tcp [ 2373.867785] Key type .llcrypt unregistered [ 2373.870961] Key type ._llcrypt unregistered [ 2404.537985] Key type ._llcrypt registered [ 2404.547039] Key type .llcrypt registered [ 2404.685996] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2405.998826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.191124] alg: No test for adler32 (adler32-zlib) [ 2407.201438] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2407.418842] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2409.079223] Key type lgssc registered [ 2410.281190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2417.989724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2418.011806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.396523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.370608] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2428.906059] LustreError: 75865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.929914] LustreError: 75865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2434.021232] LustreError: 75866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.866381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.275612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2440.459447] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2448.808676] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2456.305684] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2464.424377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2464.765703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.772865] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug +malloc all [ 2480.563585] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2489.324966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2499.551955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2499.555658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.566095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2499.577276] Lustre: Skipped 1 previous similar message [ 2499.590858] Lustre: Skipped 1 previous similar message [ 2504.672362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.679336] Lustre: Skipped 1 previous similar message [ 2505.249316] Lustre: server umount lustre-OST0000 complete [ 2509.798695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.823226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.916167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.925200] Lustre: Skipped 1 previous similar message [ 2515.595147] Lustre: server umount lustre-MDT0000 complete [ 2519.216977] LustreError: 75844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782611843 with bad export cookie 15131173074259522840 [ 2519.221592] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.230020] LustreError: 75844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2519.545788] Lustre: server umount lustre-MDT0001 complete [ 2530.743338] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2533.727688] Key type lgssc unregistered [ 2534.032055] LNet: 79421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.037628] LNetError: 79421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.081683] LNet: Removed LNI 192.168.201.139@tcp [ 2536.001376] Key type .llcrypt unregistered [ 2536.003380] Key type ._llcrypt unregistered [ 2563.619331] Key type ._llcrypt registered [ 2563.629584] Key type .llcrypt registered [ 2563.778330] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2565.518950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2565.559699] alg: No test for adler32 (adler32-zlib) [ 2566.564518] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2566.800513] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2568.479237] Key type lgssc registered [ 2570.021039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.672150] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2579.732100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.487618] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2600.609568] LustreError: 80814:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [ 2627.232966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.302625] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2642.756289] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2652.453346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.618561] Lustre: *** cfs_fail_loc=706, val=0*** [ 2653.193194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.324954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2658.272391] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2658.309376] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2658.838043] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2662.440702] LustreError: 81900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.463216] LustreError: 81900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2667.502287] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.528587] LustreError: 80839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2668.449000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.840623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.045097] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2682.890505] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2689.513706] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2694.115143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.128913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.149143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.630653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.636265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.661837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2697.965938] Lustre: server umount lustre-OST0000 complete [ 2704.865111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.868972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.901461] Lustre: Skipped 1 previous similar message [ 2704.931266] Lustre: Skipped 1 previous similar message [ 2709.989332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.998192] Lustre: Skipped 1 previous similar message [ 2714.090066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.127154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2716.456501] Lustre: server umount lustre-MDT0000 complete [ 2719.205594] LustreError: 81904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.223827] LustreError: 81904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2720.212507] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612044 with bad export cookie 8190377867914524457 [ 2720.218842] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.221683] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2720.597320] Lustre: server umount lustre-MDT0001 complete [ 2731.859815] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 2734.432511] Key type lgssc unregistered [ 2734.759762] LNet: 84434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.768282] LNetError: 84434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.791806] LNet: Removed LNI 192.168.201.139@tcp [ 2735.738176] Key type .llcrypt unregistered [ 2735.740600] Key type ._llcrypt unregistered [ 2763.875339] Key type ._llcrypt registered [ 2763.881710] Key type .llcrypt registered [ 2763.985877] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2765.599461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2765.618507] alg: No test for adler32 (adler32-zlib) [ 2766.686112] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2766.933871] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 2768.655170] Key type lgssc registered [ 2769.797502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2779.171841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2779.189620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.678987] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2796.306927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.111422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.575828] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2802.144850] LustreError: 86551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.174460] LustreError: 86551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2807.274716] LustreError: 86554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.396951] LustreError: 86555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.427650] LustreError: 86555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2812.589637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.094645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.542832] LustreError: 86550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.571900] LustreError: 86550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2820.121643] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2822.628469] LustreError: 86554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.655680] LustreError: 86554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2828.992043] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2829.012290] Lustre: MGS: Client 82f5173c-00b4-470e-8f0e-956ce5b566fb (at 0@lo) reconnecting [ 2829.065483] LustreError: 86565:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2829.396223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2829.510788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2834.129162] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2849.765626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2849.781878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2849.779962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.823588] Lustre: Skipped 1 previous similar message [ 2853.901883] Lustre: server umount lustre-OST0000 complete [ 2860.002259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2860.004607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.019361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.035881] Lustre: Skipped 1 previous similar message [ 2860.041861] Lustre: Skipped 1 previous similar message [ 2864.753715] Lustre: server umount lustre-MDT0000 complete [ 2865.134961] LustreError: 86551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.149934] LustreError: 86551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2869.999931] Lustre: server umount lustre-MDT0001 complete [ 2874.453743] Lustre: server umount MGS complete [ 2887.879861] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:03:30 (1782612210) [ 2920.086612] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 2931.636229] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 2967.263672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.488364] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2983.229119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.521889] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2983.599240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2983.684464] Lustre: lustre-MDT0001: new disk, initializing [ 2983.757372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2983.777101] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2983.785832] Lustre: 93720:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2983.798675] LustreError: 93720:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2988.045317] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 2999.412592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.783483] Lustre: lustre-OST0000: new disk, initializing [ 2999.792589] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2999.804448] Lustre: 94488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2999.915116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.147460] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3008.610085] Lustre: 94920:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c679500e x1869204255974016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782612348 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3015.733253] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.931904] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3021.944641] Lustre: MGS: Client b636e0ff-2f49-4b0c-896f-a7e0c91f52b4 (at 0@lo) reconnecting [ 3021.990907] Lustre: 92990:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3022.012639] LustreError: 93706:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3022.025378] LustreError: 93706:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3022.039044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3022.128526] Lustre: lustre-MDT0000: new disk, initializing [ 3022.224728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.248083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3022.366116] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3023.328454] Lustre: 95556:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001c914563 x1869204255980672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782612363 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3026.254241] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3028.970265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3028.982352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3032.423028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 3032.441440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3032.507366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3058.148369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3058.152743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.163977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3058.168578] Lustre: Skipped 1 previous similar message [ 3064.502046] Lustre: server umount lustre-OST0000 complete [ 3069.921922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.923302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3069.938276] Lustre: Skipped 1 previous similar message [ 3069.942158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3069.942170] Lustre: Skipped 2 previous similar messages [ 3069.987745] LustreError: Skipped 1 previous similar message [ 3075.064680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3075.077772] Lustre: Skipped 2 previous similar messages [ 3075.753708] Lustre: server umount lustre-MDT0000 complete [ 3079.208678] LustreError: 95993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.231750] LustreError: 95993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3087.235133] Lustre: server umount lustre-MDT0001 complete [ 3091.081806] Lustre: server umount MGS complete [ 3103.168877] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:07:06 (1782612426) [ 3121.838971] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3124.802713] Key type lgssc unregistered [ 3125.078787] LNet: 98057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3125.084763] LNetError: 98057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3125.102787] LNet: Removed LNI 192.168.201.139@tcp [ 3125.957861] Key type .llcrypt unregistered [ 3125.963380] Key type ._llcrypt unregistered [ 3145.317752] Key type ._llcrypt registered [ 3145.320321] Key type .llcrypt registered [ 3145.409671] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3146.814119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.848259] alg: No test for adler32 (adler32-zlib) [ 3147.903514] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3148.157068] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3149.911233] Key type lgssc registered [ 3150.937914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3158.260884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3158.272088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.618030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.399306] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3169.762852] LustreError: 99279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.776149] LustreError: 99279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3170.785808] LustreError: 99280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.760074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.091227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.456517] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3185.754958] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3191.570839] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3198.343966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3198.767587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.738283] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3204.919588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3213.666635] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3225.570797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3225.572607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.581709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3225.583560] Lustre: Skipped 1 previous similar message [ 3225.594663] Lustre: Skipped 1 previous similar message [ 3229.428930] Lustre: server umount lustre-OST0000 complete [ 3235.823329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.824229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3235.829098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.839037] Lustre: Skipped 1 previous similar message [ 3239.891585] Lustre: server umount lustre-MDT0000 complete [ 3240.934450] LustreError: 99280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3240.945158] LustreError: 99280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3243.748365] LustreError: 99260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612567 with bad export cookie 9118892188633919350 [ 3243.756705] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3243.767252] LustreError: 99260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3244.174433] Lustre: server umount lustre-MDT0001 complete [ 3254.971680] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3257.957144] Key type lgssc unregistered [ 3258.357405] LNet: 102817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3258.369409] LNetError: 102817:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3258.388876] LNet: Removed LNI 192.168.201.139@tcp [ 3259.130908] Key type .llcrypt unregistered [ 3259.134184] Key type ._llcrypt unregistered [ 3277.217721] Key type ._llcrypt registered [ 3277.220848] Key type .llcrypt registered [ 3277.349351] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3278.684822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3278.744110] alg: No test for adler32 (adler32-zlib) [ 3279.952177] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3280.236485] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3281.959236] Key type lgssc registered [ 3283.003643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3289.256203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3289.266820] LustreError: 103894:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3289.287868] LustreError: 103901:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3289.292839] LustreError: 103894:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3289.300281] LustreError: 103894:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3289.306276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.310449] LustreError: 103894:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3290.813195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.250495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3300.332884] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3300.540887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3305.518434] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3311.095428] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing lsmod [ 3313.915995] LustreError: 105104:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3313.948796] LustreError: 105108:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3313.956157] LustreError: 105104:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4965 [ 3313.965101] LustreError: 105104:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3313.969477] LustreError: 105104:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3313.974331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.252626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.360840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3320.802174] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.824474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.848929] Lustre: Skipped 1 previous similar message [ 3323.483119] Lustre: server umount lustre-MDT0000 complete [ 3324.964771] LustreError: 103939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.991798] LustreError: 103939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3327.729601] LustreError: 103916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612651 with bad export cookie 5647348568495807081 [ 3327.739583] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3327.742704] LustreError: 103916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3329.119674] LustreError: 104371:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000058c82f55 x1869204793872768/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3329.358034] Lustre: server umount lustre-MDT0001 complete [ 3344.299435] Lustre: server umount lustre-OST0000 complete [ 3354.578017] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3355.092293] LustreError: 106174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.221341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.420329] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3360.230205] LustreError: 106175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.348753] LustreError: 106174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.397916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.902530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3373.900102] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3381.632786] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3387.580456] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3395.443882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.879231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.994127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3402.480781] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3410.044901] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3423.713025] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.729939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.738883] Lustre: Skipped 3 previous similar messages [ 3423.751599] Lustre: Skipped 2 previous similar messages [ 3428.839568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.844357] Lustre: Skipped 1 previous similar message [ 3429.004922] Lustre: server umount lustre-OST0000 complete [ 3433.958562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.962697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.972453] Lustre: Skipped 1 previous similar message [ 3433.988469] Lustre: Skipped 1 previous similar message [ 3438.948920] Lustre: server umount lustre-MDT0000 complete [ 3439.074837] LustreError: 106170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.099044] LustreError: 106170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3442.454488] LustreError: 106154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612766 with bad export cookie 5647348568495808012 [ 3442.467126] LustreError: 106154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3442.474585] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.791637] Lustre: server umount lustre-MDT0001 complete [ 3452.772692] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3455.658975] Key type lgssc unregistered [ 3455.953272] LNet: 109727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3455.963822] LNetError: 109727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3455.981289] LNet: Removed LNI 192.168.201.139@tcp [ 3456.919821] Key type .llcrypt unregistered [ 3456.924173] Key type ._llcrypt unregistered [ 3483.328668] Key type ._llcrypt registered [ 3483.331550] Key type .llcrypt registered [ 3483.470983] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3484.526254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3484.587148] alg: No test for adler32 (adler32-zlib) [ 3485.686482] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3485.875762] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3487.521257] Key type lgssc registered [ 3488.851666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.269484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3498.304336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.814272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.490939] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3509.219192] LustreError: 111144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.247112] LustreError: 111144:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3513.732929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.217683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.305650] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3525.715386] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3532.373576] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3539.637375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3540.070091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.176367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3546.594207] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3553.788831] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3563.653411] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3568.952987] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3575.264565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.284787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.293872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.311532] Lustre: Skipped 1 previous similar message [ 3575.325283] Lustre: Skipped 1 previous similar message [ 3580.386897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3580.390932] Lustre: Skipped 1 previous similar message [ 3580.944134] Lustre: server umount lustre-OST0000 complete [ 3585.514303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.525583] Lustre: Skipped 1 previous similar message [ 3585.530863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3585.546479] Lustre: Skipped 1 previous similar message [ 3590.625386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.643040] Lustre: Skipped 1 previous similar message [ 3590.924502] Lustre: server umount lustre-MDT0000 complete [ 3595.514307] LustreError: 111125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782612919 with bad export cookie 17962993661275137832 [ 3595.518860] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3595.526636] LustreError: 111125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3595.900735] Lustre: server umount lustre-MDT0001 complete [ 3608.779868] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3612.147250] Key type lgssc unregistered [ 3612.531736] LNet: 114783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3612.553237] LNetError: 114783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3612.562754] LNet: Removed LNI 192.168.201.139@tcp [ 3613.263493] Key type .llcrypt unregistered [ 3613.265563] Key type ._llcrypt unregistered [ 3631.315819] Key type ._llcrypt registered [ 3631.318614] Key type .llcrypt registered [ 3631.429507] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3632.637088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3632.691738] alg: No test for adler32 (adler32-zlib) [ 3633.759716] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3633.993080] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3635.639266] Key type lgssc registered [ 3636.845667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3644.848401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3644.867314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.412128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.936980] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3656.673933] LustreError: 116008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.690326] LustreError: 116008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3659.742384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.092076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3665.768131] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3674.368613] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3680.071312] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3688.028629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.412326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.526186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3695.588425] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3704.242643] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3717.094090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3717.102977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.121153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.527141] Lustre: server umount lustre-OST0000 complete [ 3726.311804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.319429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3726.330478] Lustre: Skipped 1 previous similar message [ 3728.457059] Lustre: server umount lustre-MDT0000 complete [ 3731.433479] LustreError: 117973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3731.493186] LustreError: 117973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3732.211195] LustreError: 115987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613056 with bad export cookie 12208187818850299591 [ 3732.218741] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.220202] LustreError: 115987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3742.871285] Lustre: server umount lustre-MDT0001 complete [ 3754.387978] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3757.085335] Key type lgssc unregistered [ 3757.660281] LNet: 119543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3757.686052] LNetError: 119543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3757.722660] LNet: Removed LNI 192.168.201.139@tcp [ 3758.608703] Key type .llcrypt unregistered [ 3758.610533] Key type ._llcrypt unregistered [ 3785.398205] Key type ._llcrypt registered [ 3785.400904] Key type .llcrypt registered [ 3785.579560] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3787.004258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3787.046669] alg: No test for adler32 (adler32-zlib) [ 3788.318706] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3788.591778] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3790.327146] Key type lgssc registered [ 3791.657748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.909877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.938111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.476792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3806.600500] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3811.811472] LustreError: 120963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.820348] LustreError: 120963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3816.293211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.568594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3821.314552] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3829.656741] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3836.693934] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3845.119652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3845.507883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.532830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3852.098433] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 3860.102621] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3868.086973] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3871.978242] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3871.986169] Lustre: Skipped 2 previous similar messages [ 3874.466768] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3874.545230] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3876.739502] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3876.744209] Lustre: Skipped 2 previous similar messages [ 3884.118467] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3884.121094] Lustre: Skipped 1 previous similar message [ 3890.673276] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3890.683461] Lustre: Skipped 2 previous similar messages [ 3891.016441] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.39@tcp 192.168.0.[2,10]@tcp [ 3900.761915] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.39/24@tcp in log lustre-MDT0000 [ 3900.768616] Lustre: Skipped 2 previous similar messages [ 3906.711698] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.39/24@tcp [ 3906.717561] Lustre: Skipped 1 previous similar message [ 3916.866714] Lustre: server umount lustre-OST0000 complete [ 3922.400654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.402521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.422647] Lustre: Skipped 1 previous similar message [ 3922.439581] Lustre: Skipped 1 previous similar message [ 3927.527915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.531603] Lustre: Skipped 1 previous similar message [ 3930.768806] Lustre: server umount lustre-MDT0000 complete [ 3932.641978] LustreError: 120963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3932.665331] LustreError: 120963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3933.602123] Lustre: 120272:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782613241/real 1782613241] req@00000000c3a8416f x1869205326596480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782613257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3933.619559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.280145] LustreError: 120943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613258 with bad export cookie 10018116856450673290 [ 3934.282350] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3934.293379] LustreError: 120943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3934.709247] Lustre: server umount lustre-MDT0001 complete [ 3945.599868] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 3948.504552] Key type lgssc unregistered [ 3948.871900] LNet: 125867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3948.877525] LNetError: 125867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3948.908073] LNet: Removed LNI 192.168.201.139@tcp [ 3949.945093] Key type .llcrypt unregistered [ 3949.946940] Key type ._llcrypt unregistered [ 3983.203875] Key type ._llcrypt registered [ 3983.205769] Key type .llcrypt registered [ 3983.306287] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 3984.165938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3984.190236] alg: No test for adler32 (adler32-zlib) [ 3985.303357] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3985.511261] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 3987.161377] Key type lgssc registered [ 3988.070625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3995.947839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3995.972425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.473193] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3997.494243] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.39/24@tcp [ 3997.505365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.257184] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4006.896987] LustreError: 127430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.920328] LustreError: 127430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4012.016980] LustreError: 127431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4012.217796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.454401] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4012.477200] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.39/24@tcp [ 4012.532703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4017.330587] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4024.567488] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4030.421067] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4037.830284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.374111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.481774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4045.322415] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4052.856880] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4068.327119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4068.328077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4068.340822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4068.353614] Lustre: Skipped 1 previous similar message [ 4073.446366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4073.448260] Lustre: Skipped 1 previous similar message [ 4074.312862] Lustre: server umount lustre-OST0000 complete [ 4078.567617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.573442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4078.583031] Lustre: Skipped 1 previous similar message [ 4078.600536] Lustre: Skipped 1 previous similar message [ 4083.683223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.695118] Lustre: Skipped 1 previous similar message [ 4084.749251] Lustre: server umount lustre-MDT0000 complete [ 4088.806599] LustreError: 127430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.832550] LustreError: 127430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4089.450100] LustreError: 129282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613413 with bad export cookie 1618392600921256224 [ 4089.462276] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.468301] LustreError: 129282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4090.037559] Lustre: server umount lustre-MDT0001 complete [ 4103.868729] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4107.483278] Key type lgssc unregistered [ 4107.692899] LNet: 131019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4107.700783] LNetError: 131019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4107.712936] LNet: Removed LNI 192.168.201.139@tcp [ 4108.691782] Key type .llcrypt unregistered [ 4108.693572] Key type ._llcrypt unregistered [ 4141.960670] Key type ._llcrypt registered [ 4141.963247] Key type .llcrypt registered [ 4142.139462] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4162.967978] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4164.189892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4164.255556] alg: No test for adler32 (adler32-zlib) [ 4165.381222] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4165.712402] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4167.511547] Key type lgssc registered [ 4168.718678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4203.524992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4203.573732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4204.823633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4204.866918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4204.965444] Lustre: lustre-MDT0000: new disk, initializing [ 4205.067507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4205.088076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4210.195389] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4220.554838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.688482] Lustre: 134422:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4220.749334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4220.752948] Lustre: Skipped 1 previous similar message [ 4220.822592] Lustre: lustre-MDT0001: new disk, initializing [ 4220.890377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4220.934415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4220.948833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4225.619599] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4236.525059] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4243.147414] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4252.476885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.863360] Lustre: lustre-OST0000: new disk, initializing [ 4252.873650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4252.890875] Lustre: 136390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.993551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.333313] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4261.958230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4261.968050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4262.147643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4270.636479] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4275.540582] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4275.765290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4280.191095] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4280.461696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.848933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.857709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.869055] Lustre: Skipped 1 previous similar message [ 4282.896536] Lustre: Skipped 1 previous similar message [ 4287.975148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4287.992548] Lustre: Skipped 1 previous similar message [ 4288.736626] Lustre: server umount lustre-OST0000 complete [ 4292.578406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.587783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.617871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.215068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.227663] Lustre: Skipped 3 previous similar messages [ 4298.859516] Lustre: server umount lustre-MDT0000 complete [ 4303.353034] LustreError: 134416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613627 with bad export cookie 2203372270539706776 [ 4303.357136] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4303.364953] LustreError: 134416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4303.931476] Lustre: server umount lustre-MDT0001 complete [ 4314.691485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.263352] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.280920] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4315.344607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.111700] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4320.739527] LustreError: 138611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.859571] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4329.409142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.900304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4335.197649] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4344.848105] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4351.727315] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4360.371555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.843063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.606941] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4376.699640] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4381.882517] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4382.223713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4387.427177] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4387.691561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4396.766557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.968457] Lustre: lustre-OST0001: new disk, initializing [ 4396.972565] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4396.979089] Lustre: 141986:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4397.075499] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4403.346845] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4403.777860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4403.788780] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4403.895677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4411.930109] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4412.139402] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4417.268457] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4417.632855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4424.285318] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c3c31d2800.ost_server_uuid 50 [ 4426.489389] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c3c31d2800.ost_server_uuid in IDLE state after 0 sec [ 4440.033125] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.040341] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.043727] LustreError: Skipped 1 previous similar message [ 4440.045477] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4440.080719] Lustre: Skipped 2 previous similar messages [ 4450.277690] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4450.281693] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4450.318521] Lustre: Skipped 4 previous similar messages [ 4450.499383] Lustre: server umount lustre-OST0001 complete [ 4455.398359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.414394] LustreError: 140433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.439220] LustreError: 140433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4460.517876] LustreError: 140432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4460.554396] LustreError: 140432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4461.095798] Lustre: server umount lustre-OST0000 complete [ 4465.643097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.670177] Lustre: Skipped 2 previous similar messages [ 4470.753513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.766577] Lustre: Skipped 7 previous similar messages [ 4472.081739] Lustre: server umount lustre-MDT0000 complete [ 4475.876240] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.918475] LustreError: 138606:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4476.200692] LustreError: 141985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782613800 with bad export cookie 2203372270539707658 [ 4476.201764] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.220203] LustreError: 141985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4476.628191] Lustre: server umount lustre-MDT0001 complete [ 4500.925239] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4504.229281] Key type lgssc unregistered [ 4504.582518] LNet: 144990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4504.588795] LNetError: 144990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4504.610205] LNet: Removed LNI 192.168.201.139@tcp [ 4505.574181] Key type .llcrypt unregistered [ 4505.576062] Key type ._llcrypt unregistered [ 4554.968280] Key type ._llcrypt registered [ 4554.970685] Key type .llcrypt registered [ 4555.096234] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4571.920210] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4572.933286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4572.963300] alg: No test for adler32 (adler32-zlib) [ 4574.169761] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4574.518990] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4576.250752] Key type lgssc registered [ 4577.857526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4619.236904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4619.267572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.653214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4620.681195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4620.771919] Lustre: lustre-MDT0000: new disk, initializing [ 4620.855545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.867767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4626.191913] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4638.534698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.662788] Lustre: 149151:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4638.702651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4638.705761] Lustre: Skipped 1 previous similar message [ 4638.810902] Lustre: lustre-MDT0001: new disk, initializing [ 4638.925725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4638.962383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4638.975686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4643.611768] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4653.828692] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4660.828681] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4669.009152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4669.428578] Lustre: lustre-OST0000: new disk, initializing [ 4669.432853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4669.440390] Lustre: 151120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4669.548465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.044358] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4679.796695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4679.811224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4679.955827] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4686.703799] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4692.383993] Lustre: Failing over lustre-OST0000 [ 4693.999676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4694.024381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.050681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4694.652777] Lustre: server umount lustre-OST0000 complete [ 4710.879474] Lustre: 147372:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782614019/real 1782614019] req@00000000ee6f79fc x1869206150744064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782614035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4710.888112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.911199] Lustre: 147372:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4714.415171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.893263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4714.921461] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4715.030277] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4715.951441] Lustre: 147372:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782614024/real 1782614024] req@0000000076344b83 x1869206150745344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782614040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4721.037177] Lustre: 147373:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782614029/real 1782614029] req@00000000ae873c33 x1869206150746240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782614045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4723.057298] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4727.948076] Lustre: Failing over lustre-MDT0000 [ 4728.607428] Lustre: server umount lustre-MDT0000 complete [ 4730.403564] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.425877] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4731.361219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4731.363866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.369700] LustreError: 151333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4731.438605] Lustre: Skipped 2 previous similar messages [ 4735.528936] LustreError: 151333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.557924] LustreError: 151333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4740.613040] LustreError: 149157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.630406] LustreError: 149157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4745.719746] LustreError: 149157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.755858] LustreError: 149157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4747.744261] Lustre: 147375:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782614055/real 1782614055] req@0000000011e6fa84 x1869206150757248/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782614071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4747.795515] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.665895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.974086] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.009416] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4757.227698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4757.319673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.393614] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4761.092439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4762.606325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4762.629846] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4762.727356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4763.530747] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4777.954235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.956312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4777.973617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4777.983866] LustreError: Skipped 1 previous similar message [ 4778.003767] Lustre: Skipped 1 previous similar message [ 4779.723461] Lustre: server umount lustre-OST0000 complete [ 4788.206319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.206606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.210389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4788.238947] Lustre: Skipped 2 previous similar messages [ 4789.820201] Lustre: server umount lustre-MDT0000 complete [ 4793.314253] LustreError: 149158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.328000] LustreError: 149158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4794.092965] LustreError: 151131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782614118 with bad export cookie 7983734517781817410 [ 4794.097392] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.114280] LustreError: 151131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4794.556598] Lustre: server umount lustre-MDT0001 complete [ 4806.745705] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 4810.209366] Key type lgssc unregistered [ 4810.439185] LNet: 154785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4810.448925] LNetError: 154785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4811.493858] LNet: Removed LNI 192.168.201.139@tcp [ 4812.444493] Key type .llcrypt unregistered [ 4812.453620] Key type ._llcrypt unregistered [ 4852.340695] Key type ._llcrypt registered [ 4852.342784] Key type .llcrypt registered [ 4852.474500] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4854.311539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4854.384416] alg: No test for adler32 (adler32-zlib) [ 4855.587991] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4856.033109] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 4857.839194] Key type lgssc registered [ 4859.206752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4868.612778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4868.653817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.276408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.233064] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4880.867736] LustreError: 156308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.892765] LustreError: 156308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4885.897808] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4885.988156] LustreError: 156309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.149049] LustreError: 156337:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4890.159390] LustreError: 156337:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4890.165768] LustreError: 156337:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4890.498871] Lustre: server umount lustre-MDT0000 complete [ 4908.930905] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 4924.448564] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 4967.809932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4968.209959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4968.296069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4968.426238] Lustre: lustre-MDT0000: new disk, initializing [ 4968.596217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.628099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4973.503738] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 4985.158229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4985.281777] Lustre: 159662:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4985.329712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4985.333251] Lustre: Skipped 1 previous similar message [ 4985.400135] Lustre: lustre-MDT0001: new disk, initializing [ 4985.529725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4985.575258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4985.585217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4990.539839] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5000.428200] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5006.219697] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5014.248148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.619179] Lustre: lustre-OST0000: new disk, initializing [ 5014.622810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5014.627415] Lustre: 161630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5014.759657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.315168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5022.327222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5022.482908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5023.398340] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5034.207506] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5039.764850] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5041.680783] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:39:24 (1782614364) [ 5323.743817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5323.751598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.769704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.768166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5324.768790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5324.785486] Lustre: Skipped 1 previous similar message [ 5324.799750] Lustre: Skipped 2 previous similar messages [ 5329.027824] Lustre: server umount lustre-MDT0000 complete [ 5329.894038] LustreError: 163406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.921143] LustreError: 163406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5333.667930] LustreError: 163384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782614657 with bad export cookie 1190754878495160510 [ 5333.681599] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.687921] LustreError: 163384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5335.007916] LustreError: 159669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.011496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5335.050397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5338.017278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5340.275649] Lustre: server umount lustre-MDT0001 complete [ 5351.547895] Lustre: server umount lustre-OST0000 complete [ 5362.086362] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5373.275473] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5411.006824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.299719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.327601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.457905] Lustre: lustre-MDT0000: new disk, initializing [ 5411.600561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.623978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5416.666986] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5427.103578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.227979] Lustre: 166305:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5427.278224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5427.281975] Lustre: Skipped 1 previous similar message [ 5427.361991] Lustre: lustre-MDT0001: new disk, initializing [ 5427.450389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5427.481685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5427.492774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5432.254552] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5443.255043] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.346357] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5457.512654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5457.866866] Lustre: lustre-OST0000: new disk, initializing [ 5457.869516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5457.882239] Lustre: 168273:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5458.025712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.787596] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5465.113656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5465.140951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5465.285077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5475.493257] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5480.504188] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5480.755186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5485.585457] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5485.806218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5491.169922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5491.172576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5491.214845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5491.216425] LustreError: Skipped 1 previous similar message [ 5491.231356] Lustre: Skipped 1 previous similar message [ 5494.061752] Lustre: server umount lustre-OST0000 complete [ 5499.369581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5499.377799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.389064] Lustre: Skipped 1 previous similar message [ 5499.400336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.687223] Lustre: server umount lustre-MDT0000 complete [ 5506.542425] LustreError: 166312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.562241] LustreError: 166312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5509.372654] LustreError: 166295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782614833 with bad export cookie 1190754878495319333 [ 5509.381360] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.387584] LustreError: 166295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5509.819628] Lustre: server umount lustre-MDT0001 complete [ 5519.507751] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:47:22 (1782614842) [ 5539.375640] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5550.763548] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5597.793656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.180050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5598.232659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5598.398772] Lustre: lustre-MDT0000: new disk, initializing [ 5598.558588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5598.593737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5606.756953] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5619.268224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.525315] Lustre: 173031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5619.982715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5625.316630] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5635.648304] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5642.944740] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5650.977753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.258236] Lustre: lustre-OST0000: new disk, initializing [ 5651.261606] Lustre: Skipped 1 previous similar message [ 5651.272112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5651.281804] Lustre: Skipped 2 previous similar messages [ 5651.291749] Lustre: 175011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5657.221526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5657.235773] Lustre: Skipped 1 previous similar message [ 5657.249474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5657.472842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5659.046928] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5670.295473] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5678.049605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.050553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.052406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5678.052415] Lustre: Skipped 2 previous similar messages [ 5678.075063] Lustre: Skipped 2 previous similar messages [ 5678.088541] LustreError: Skipped 1 previous similar message [ 5683.761749] Lustre: server umount lustre-OST0000 complete [ 5691.364925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5695.077858] Lustre: server umount lustre-MDT0000 complete [ 5698.528528] LustreError: 173040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5698.550162] LustreError: 173040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5699.103611] LustreError: 175010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615023 with bad export cookie 1190754878495320222 [ 5699.106892] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.113341] LustreError: 175010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5699.654269] Lustre: server umount lustre-MDT0001 complete [ 5710.477623] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:50:33 (1782615033) [ 5730.395152] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 5743.366982] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5787.999654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.340638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5788.385766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5788.480573] Lustre: lustre-MDT0000: new disk, initializing [ 5788.608645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5788.614224] Lustre: Skipped 2 previous similar messages [ 5788.636699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5794.844455] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5808.213676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.479391] Lustre: 179339:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5809.019263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5814.190077] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5825.809805] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.821080] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5842.448213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.037745] Lustre: 181316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5849.692435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5849.845035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5850.505437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5861.075626] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5870.565789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.574199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5870.581819] Lustre: Skipped 2 previous similar messages [ 5870.584148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5870.591064] LustreError: Skipped 1 previous similar message [ 5870.610262] Lustre: Skipped 7 previous similar messages [ 5876.402816] Lustre: server umount lustre-OST0000 complete [ 5895.135230] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5895.480405] Lustre: server umount lustre-MDT0000 complete [ 5896.163040] LustreError: 179344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.191796] LustreError: 179344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5900.124850] LustreError: 179331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615224 with bad export cookie 1190754878495321307 [ 5900.133378] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5900.159749] LustreError: 179331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5900.676573] Lustre: server umount lustre-MDT0001 complete [ 5914.509988] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 5917.475057] Key type lgssc unregistered [ 5917.694854] LNet: 183174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5917.700602] LNetError: 183174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5918.759032] LNet: Removed LNI 192.168.201.139@tcp [ 5919.767213] Key type .llcrypt unregistered [ 5919.769670] Key type ._llcrypt unregistered [ 5950.198885] Key type ._llcrypt registered [ 5950.202501] Key type .llcrypt registered [ 5950.310510] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 5951.768902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5951.863695] alg: No test for adler32 (adler32-zlib) [ 5953.053075] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5953.465098] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 5955.207301] Key type lgssc registered [ 5956.564719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5965.413060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5965.425569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.196736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5972.698798] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5977.570932] LustreError: 184592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.591564] LustreError: 184592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5982.695255] LustreError: 184591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.960066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.412394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5988.462467] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 5997.669967] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6004.387229] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6011.930513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.466303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6017.553240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6019.890987] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6028.324695] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6049.258760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6049.265199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6049.265806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6049.274687] Lustre: Skipped 1 previous similar message [ 6054.389016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.400756] Lustre: Skipped 2 previous similar messages [ 6055.155715] Lustre: server umount lustre-OST0000 complete [ 6064.609077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6064.610664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6064.644321] Lustre: Skipped 1 previous similar message [ 6064.657697] Lustre: Skipped 1 previous similar message [ 6066.370474] Lustre: server umount lustre-MDT0000 complete [ 6069.730162] LustreError: 184591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.639350] LustreError: 184571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615394 with bad export cookie 12174618266816256382 [ 6070.655742] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6070.658889] LustreError: 184571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6071.046318] Lustre: server umount lustre-MDT0001 complete [ 6090.120528] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 6093.836767] Key type lgssc unregistered [ 6094.236711] LNet: 188168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6094.249716] LNetError: 188168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6094.272953] LNet: Removed LNI 192.168.201.139@tcp [ 6095.559671] Key type .llcrypt unregistered [ 6095.561873] Key type ._llcrypt unregistered [ 6124.048291] Key type ._llcrypt registered [ 6124.049930] Key type .llcrypt registered [ 6124.198689] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6125.929848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6126.090676] alg: No test for adler32 (adler32-zlib) [ 6127.287274] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 6127.869827] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 6129.632565] Key type lgssc registered [ 6131.726631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6143.514355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.557237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.206133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.334765] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6155.749175] LustreError: 189585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.774493] LustreError: 189585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6160.511699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.827561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6166.326443] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6175.009586] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.921585] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6190.840761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.248678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.345840] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6210.150231] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6215.889302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6226.408882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.420794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.423812] LustreError: Skipped 1 previous similar message [ 6226.426504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.446613] Lustre: Skipped 1 previous similar message [ 6227.514857] Lustre: lustre-OST0000: Not available for connect from 192.168.201.39@tcp (stopping) [ 6227.525383] Lustre: Skipped 1 previous similar message [ 6229.041763] Lustre: server umount lustre-OST0000 complete [ 6239.161871] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90c3c81ed800.ost_server_uuid 50 [ 6241.861520] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6258.149215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6258.166937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.233918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6261.218240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6261.229499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.057333] Lustre: server umount lustre-MDT0000 complete [ 6266.352737] LustreError: 189586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.386779] LustreError: 189586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6267.360716] LustreError: 189582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.375251] LustreError: 189582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6267.490894] LustreError: 189566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615591 with bad export cookie 18155461744801441123 [ 6267.501679] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.509541] LustreError: 189566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6267.984787] Lustre: server umount lustre-MDT0001 complete [ 6279.151663] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:00:01 (1782615601) [ 6288.461946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.197981] LustreError: 193504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.360408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.499375] LustreError: 193505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.769775] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6304.741227] LustreError: 193505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.776689] LustreError: 193505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6306.049727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.651584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6313.230206] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6324.091566] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6332.054955] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6339.687528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6340.098719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.018198] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6355.589764] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6359.567623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6363.784240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.041343] Lustre: lustre-OST0001: new disk, initializing [ 6364.044788] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6364.051427] Lustre: 196362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6364.159274] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6370.875400] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6371.973284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6371.987825] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6372.174216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6382.121217] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6385.125796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6385.133834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.157828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6385.168893] Lustre: Skipped 1 previous similar message [ 6387.169624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.440683] Lustre: server umount lustre-OST0000 complete [ 6392.819439] LustreError: 195339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.856832] LustreError: 195339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6398.275871] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6398.542564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6403.455553] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6403.695151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6423.533052] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6423.536218] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6423.545727] LustreError: Skipped 1 previous similar message [ 6423.558916] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6423.563434] Lustre: Skipped 2 previous similar messages [ 6423.579206] Lustre: Skipped 4 previous similar messages [ 6424.698025] Lustre: server umount lustre-OST0001 complete [ 6433.760685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6433.771341] Lustre: Skipped 1 previous similar message [ 6435.359391] Lustre: server umount lustre-MDT0000 complete [ 6438.891275] LustreError: 197170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6438.947688] LustreError: 197170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 6439.699087] LustreError: 193483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615763 with bad export cookie 18155461744801442194 [ 6439.720979] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.311530] Lustre: server umount lustre-MDT0001 complete [ 6467.120761] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:03:10 (1782615790) [ 6474.816665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.088423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6475.133469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6475.329232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.579200] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6489.477196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.573542] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6489.658627] Lustre: 199804:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6495.724220] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6503.963053] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6510.124816] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6518.109180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6518.269937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6525.679697] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6528.216499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6534.531037] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6542.140871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.195535] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6542.401252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6542.407737] Lustre: Skipped 2 previous similar messages [ 6549.567097] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6550.250632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6557.562919] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6560.750445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6560.752806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6560.760041] Lustre: Skipped 1 previous similar message [ 6560.768749] Lustre: Skipped 4 previous similar messages [ 6567.191675] Lustre: server umount lustre-OST0000 complete [ 6568.457120] LustreError: 201656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.326769] Lustre: server umount lustre-OST0001 complete [ 6596.592196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.601838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.615525] Lustre: Skipped 1 previous similar message [ 6596.634858] Lustre: Skipped 4 previous similar messages [ 6604.963972] Lustre: server umount lustre-MDT0000 complete [ 6607.839776] Lustre: 188898:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782615915/real 1782615915] req@000000001158d1b7 x1869207780368000/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782615931 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6609.567407] LustreError: 200551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782615933 with bad export cookie 18155461744801443601 [ 6609.570661] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6609.577701] LustreError: 200551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6609.898365] Lustre: server umount lustre-MDT0001 complete [ 6638.013586] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:06:00 (1782615960) [ 6656.145706] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 6668.982857] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 6711.760893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.146215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6712.175610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6712.264412] Lustre: lustre-MDT0000: new disk, initializing [ 6712.336188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6712.356502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6717.024188] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6728.213862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.361878] Lustre: 207903:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6728.422971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6728.426967] Lustre: Skipped 1 previous similar message [ 6728.528971] Lustre: lustre-MDT0001: new disk, initializing [ 6728.653576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6728.670722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6733.722933] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6745.430511] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6753.208278] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6760.986866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.466053] Lustre: lustre-OST0000: new disk, initializing [ 6761.472574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6761.482959] Lustre: 209871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6769.172717] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6770.278492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6770.290576] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6770.535951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6779.955717] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6785.751273] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6785.984557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6791.202313] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6791.433081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6796.259972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.267671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6796.279795] Lustre: Skipped 2 previous similar messages [ 6796.287609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6796.294410] LustreError: Skipped 1 previous similar message [ 6796.314650] Lustre: Skipped 3 previous similar messages [ 6800.003738] Lustre: server umount lustre-OST0000 complete [ 6810.924329] Lustre: server umount lustre-MDT0000 complete [ 6811.616087] LustreError: 207914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6811.644314] LustreError: 207914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 6814.507748] LustreError: 207893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616138 with bad export cookie 18155461744801445379 [ 6814.512281] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6814.514566] LustreError: 207893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6814.944298] Lustre: server umount lustre-MDT0001 complete [ 6824.469584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6830.571669] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6839.977343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.814783] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6856.388703] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6863.214644] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6871.613339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6879.544760] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6888.366577] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6893.421832] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6896.942834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6902.021631] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6902.331452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6906.349733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6906.363865] LustreError: Skipped 2 previous similar messages [ 6910.398467] Lustre: server umount lustre-OST0000 complete [ 6918.164850] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6924.083770] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6955.548665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6964.029577] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 6971.371826] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6971.385022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6971.388074] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6974.309539] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6976.184069] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6981.602434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.606347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6981.606573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6981.624524] Lustre: Skipped 5 previous similar messages [ 6981.640310] Lustre: Skipped 7 previous similar messages [ 6986.404848] Lustre: server umount lustre-OST0000 complete [ 6997.579671] Lustre: server umount lustre-MDT0000 complete [ 7002.471561] LustreError: 212071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616326 with bad export cookie 18155461744801446268 [ 7002.474280] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7002.481509] LustreError: 212071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7002.882503] Lustre: server umount lustre-MDT0001 complete [ 7013.808804] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:12:16 (1782616336) [ 7022.282208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.685750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7022.694975] Lustre: Skipped 6 previous similar messages [ 7028.072319] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7038.237586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.186129] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7053.361794] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7059.710939] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7068.306637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.467509] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7084.588992] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7089.792294] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7093.253803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7098.282965] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7098.531591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7105.985932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7106.162587] Lustre: lustre-OST0001: new disk, initializing [ 7106.166300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7106.174232] Lustre: 221482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7112.822535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7112.849032] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7113.013589] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7113.348684] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7123.932765] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7129.391666] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7129.666536] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7135.797143] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7136.076820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7138.784812] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.798722] LustreError: Skipped 1 previous similar message [ 7153.122194] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7153.375204] Lustre: server umount lustre-OST0001 complete [ 7154.158451] LustreError: 221807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.182972] LustreError: 221807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 7160.926923] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7161.232809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7167.116367] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7167.380984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7197.696793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7203.319360] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7203.352858] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7203.354180] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7203.390349] Lustre: Skipped 2 previous similar messages [ 7205.583460] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7215.881882] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7218.003482] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7227.012771] Lustre: server umount lustre-OST0001 complete [ 7246.308305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.328414] Lustre: Skipped 6 previous similar messages [ 7246.340447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7246.353410] Lustre: Skipped 12 previous similar messages [ 7252.338935] Lustre: server umount lustre-OST0000 complete [ 7262.808050] Lustre: server umount lustre-MDT0000 complete [ 7267.093326] LustreError: 218068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616591 with bad export cookie 18155461744801447234 [ 7267.108679] LustreError: 218068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7267.111846] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7273.520413] Lustre: server umount lustre-MDT0001 complete [ 7300.489731] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:17:03 (1782616623) [ 7308.446269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.676804] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7308.709592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7313.997639] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7324.763166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.938972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7324.993806] Lustre: 226731:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7331.175513] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7341.924788] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7350.843097] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7359.550695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.845755] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7366.708942] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7375.292434] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7385.542525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.620440] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7392.354365] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7400.898572] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7403.128498] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7408.538322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7408.731696] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7415.014671] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7415.340357] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7423.652420] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c3c81ed800.ost_server_uuid 50 [ 7425.905278] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90c3c81ed800.ost_server_uuid in IDLE state after 0 sec [ 7426.927314] Lustre: Permanently deactivating lustre-OST0001 [ 7426.944760] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7431.366737] Lustre: Permanently reactivating lustre-OST0001 [ 7431.416566] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7431.422365] Lustre: Skipped 2 previous similar messages [ 7441.358410] Lustre: server umount lustre-OST0001 complete [ 7441.409725] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7441.421166] LustreError: Skipped 5 previous similar messages [ 7460.320789] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7460.559815] Lustre: server umount lustre-OST0000 complete [ 7470.755330] Lustre: server umount lustre-MDT0000 complete [ 7474.740948] LustreError: 226724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616798 with bad export cookie 18155461744801448522 [ 7474.744810] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7474.751377] LustreError: 226724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7474.978672] Lustre: server umount lustre-MDT0001 complete [ 7502.935722] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:20:25 (1782616825) [ 7511.618632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.978352] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7512.016699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7512.027984] Lustre: Skipped 2 previous similar messages [ 7516.963248] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7526.019474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7526.106287] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7526.130519] Lustre: 233428:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7531.693210] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7539.940708] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7546.538013] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7554.504765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.763139] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7554.806433] Lustre: Permanently deactivating lustre-OST0000 [ 7554.834520] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7555.033258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7555.049338] Lustre: Skipped 10 previous similar messages [ 7562.806910] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7564.916625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7571.481645] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7580.250915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.307502] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7589.187105] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7589.674549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7597.949391] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7599.926136] Lustre: Permanently reactivating lustre-OST0000 [ 7599.953185] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7599.963103] Lustre: Skipped 2 previous similar messages [ 7606.901073] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7606.902278] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 0000000034c91dcb, cur 1782616931 deadline 1782616919 last 1782616889 [ 7606.911360] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7606.926040] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7606.970706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7631.327728] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7631.402361] Lustre: server umount lustre-OST0001 complete [ 7641.621585] Lustre: server umount lustre-OST0000 complete [ 7651.777651] Lustre: server umount lustre-MDT0000 complete [ 7655.050689] LustreError: 233421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782616979 with bad export cookie 18155461744801450216 [ 7655.067021] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.080639] LustreError: 233421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7655.738885] Lustre: server umount lustre-MDT0001 complete [ 7668.458690] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 7671.064957] Key type lgssc unregistered [ 7671.375878] LNet: 238313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7671.384046] LNetError: 238313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7671.402894] LNet: Removed LNI 192.168.201.139@tcp [ 7672.378599] Key type .llcrypt unregistered [ 7672.380952] Key type ._llcrypt unregistered [ 7696.706395] Key type ._llcrypt registered [ 7696.708669] Key type .llcrypt registered [ 7696.813175] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7697.720032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7697.817779] alg: No test for adler32 (adler32-zlib) [ 7698.892705] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 7699.175449] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 7700.928154] Key type lgssc registered [ 7702.024362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7712.867622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7712.887239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.311115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.957942] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7724.513935] LustreError: 239780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7724.543746] LustreError: 239780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7728.689947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.786242] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7728.812989] Lustre: 239770:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7728.827694] Lustre: Permanently deactivating lustre-MDT0001 [ 7728.841408] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7729.200959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7734.143834] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7738.910092] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7742.930410] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7749.106485] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7757.393156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.778589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.860033] LustreError: 241623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7759.880450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7764.816515] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7764.962171] LustreError: 241623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7770.100824] LustreError: 241624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.235601] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7774.195908] LustreError: 241732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.227710] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7774.233047] LustreError: 241732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7781.264815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7781.540356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7787.038433] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7787.038291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7788.084376] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7795.856881] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7799.128982] Lustre: Permanently reactivating lustre-MDT0001 [ 7799.148639] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7799.154812] Lustre: Skipped 2 previous similar messages [ 7804.986354] Lustre: lustre-MDT0001: Client 99a996f4-3067-431c-956e-d729c5ad6e70 (at 192.168.201.39@tcp) reconnecting [ 7809.681704] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7809.692725] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7809.699750] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7809.709486] LustreError: 243496:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7809.722481] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7809.723317] LustreError: 243496:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7809.723598] LustreError: 243496:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.139@tcp: cfg command failed: rc = -108 [ 7809.723622] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7809.723622] [ 7809.724051] LustreError: 239772:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7812.014875] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7814.208840] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7822.410764] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7822.871035] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7828.878538] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7829.060665] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7831.927403] Lustre: Permanently deactivating lustre-MDT0001 [ 7831.943888] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7831.950879] Lustre: Skipped 2 previous similar messages [ 7839.350951] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7846.241659] Lustre: server umount lustre-MDT0000 complete [ 7846.880570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7846.885874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7846.906120] LustreError: 240790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.949426] LustreError: 240790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7848.432968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.446774] Lustre: Skipped 1 previous similar message [ 7850.601336] LustreError: 239761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617174 with bad export cookie 10173246595548957229 [ 7850.604302] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7850.616687] LustreError: 239761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7851.281816] Lustre: server umount lustre-MDT0001 complete [ 7857.037753] Lustre: server umount lustre-OST0000 complete [ 7872.707673] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:26:34 (1782617194) [ 7884.196191] Lustre: server umount lustre-OST0001 complete [ 7890.483859] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 7900.077450] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 7942.362317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.746234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7942.753560] Lustre: Skipped 2 previous similar messages [ 7942.808538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7942.920244] Lustre: lustre-MDT0000: new disk, initializing [ 7943.026607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7943.048668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7947.411515] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7958.220580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.383626] Lustre: 248058:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7958.482284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7958.488680] Lustre: Skipped 1 previous similar message [ 7958.596350] Lustre: lustre-MDT0001: new disk, initializing [ 7958.717682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7958.754031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7958.775528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7963.471185] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7973.139254] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7979.114670] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7985.191362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7985.442199] Lustre: lustre-OST0000: new disk, initializing [ 7985.445952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7985.452417] Lustre: 250029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7985.524964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.644051] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 7994.442879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7994.459593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7994.601677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8001.172890] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8003.675602] LustreError: 248066:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8013.121336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.386416] Lustre: lustre-OST0001: new disk, initializing [ 8013.391696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8013.399297] Lustre: 251144:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8013.703083] LustreError: 248066:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8013.768423] LustreError: 248065:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8019.045463] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8022.721390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8022.748373] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8022.893775] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8023.799663] LustreError: 248065:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8029.188820] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8030.995566] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8033.251679] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.255594] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.264089] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8033.267480] Lustre: Skipped 2 previous similar messages [ 8033.286314] Lustre: Skipped 1 previous similar message [ 8038.380707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8038.386884] Lustre: Skipped 2 previous similar messages [ 8039.520680] Lustre: server umount lustre-OST0001 complete [ 8043.501158] LustreError: 250493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8043.516687] LustreError: 250493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8058.856848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8058.859881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8058.867502] Lustre: Skipped 1 previous similar message [ 8058.874463] Lustre: Skipped 2 previous similar messages [ 8063.025099] Lustre: server umount lustre-OST0000 complete [ 8069.089985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.102131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8069.107137] Lustre: Skipped 1 previous similar message [ 8069.122410] Lustre: Skipped 1 previous similar message [ 8074.217202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8074.224085] Lustre: Skipped 1 previous similar message [ 8081.376747] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8081.720706] Lustre: server umount lustre-MDT0000 complete [ 8084.448901] LustreError: 248071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8084.463291] LustreError: 248071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 8085.689706] LustreError: 248051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617409 with bad export cookie 10173246595548960043 [ 8085.692875] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8085.700024] LustreError: 248051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8100.319151] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8100.802843] Lustre: server umount lustre-MDT0001 complete [ 8113.769513] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8116.869592] Key type lgssc unregistered [ 8117.163910] LNet: 253179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8117.176687] LNetError: 253179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8117.201311] LNet: Removed LNI 192.168.201.139@tcp [ 8118.149317] Key type .llcrypt unregistered [ 8118.154167] Key type ._llcrypt unregistered [ 8156.670392] Key type ._llcrypt registered [ 8156.679371] Key type .llcrypt registered [ 8156.782909] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8157.963104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8158.165941] alg: No test for adler32 (adler32-zlib) [ 8159.256512] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8159.489380] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8161.209425] Key type lgssc registered [ 8162.454292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8170.301501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8170.316550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.570250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8171.608049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8171.857207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.114666] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8186.605654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8186.675084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8186.699350] Lustre: 255387:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8186.888947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8191.721413] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8199.262956] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8207.672574] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8216.308897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8216.515221] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8216.685877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.475798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8224.873481] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8233.895855] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8294.377224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8294.390620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.391809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8294.407715] Lustre: Skipped 1 previous similar message [ 8296.460923] Lustre: server umount lustre-OST0000 complete [ 8301.177558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8315.013791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8315.496911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.874635] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8330.910844] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8334.890567] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8334.911041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8334.911648] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8334.941432] Lustre: Skipped 1 previous similar message [ 8375.783282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8375.796468] LustreError: Skipped 1 previous similar message [ 8375.799781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.811221] Lustre: Skipped 1 previous similar message [ 8375.818851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8375.837532] Lustre: Skipped 2 previous similar messages [ 8377.828308] Lustre: server umount lustre-OST0000 complete [ 8386.017107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8386.036220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.043496] Lustre: Skipped 1 previous similar message [ 8386.056741] Lustre: Skipped 1 previous similar message [ 8391.138579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.152628] Lustre: Skipped 1 previous similar message [ 8396.256965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8396.258037] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 8396.276614] Lustre: Skipped 1 previous similar message [ 8396.624413] Lustre: server umount lustre-MDT0000 complete [ 8400.912929] LustreError: 255379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617725 with bad export cookie 5417994047726241883 [ 8400.920456] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8400.921054] LustreError: 255379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8401.249409] Lustre: server umount lustre-MDT0001 complete [ 8412.083476] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8414.671389] Key type lgssc unregistered [ 8414.903097] LNet: 260815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8414.909518] LNetError: 260815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8414.926752] LNet: Removed LNI 192.168.201.139@tcp [ 8415.678514] Key type .llcrypt unregistered [ 8415.680760] Key type ._llcrypt unregistered [ 8440.177337] Key type ._llcrypt registered [ 8440.180843] Key type .llcrypt registered [ 8440.354578] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8441.519744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8441.683477] alg: No test for adler32 (adler32-zlib) [ 8442.739078] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8442.969052] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8444.681963] Key type lgssc registered [ 8445.847848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8454.194772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8454.215351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.642544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8460.619468] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8464.868699] LustreError: 262238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.906385] LustreError: 262238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8469.985980] LustreError: 262237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8470.199973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8470.793854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8475.295593] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8483.076434] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8488.672229] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8496.173475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8496.531718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8500.674379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8503.232503] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8513.110606] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8541.668366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.686890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.717901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.781475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8545.796721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8545.804558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.077937] Lustre: server umount lustre-OST0000 complete [ 8550.886629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.913060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.926903] Lustre: Skipped 2 previous similar messages [ 8556.005600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.016336] Lustre: Skipped 1 previous similar message [ 8557.199577] Lustre: server umount lustre-MDT0000 complete [ 8561.021665] LustreError: 262218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782617885 with bad export cookie 9092308654207452552 [ 8561.028172] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8561.042609] LustreError: 262218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8561.657547] Lustre: server umount lustre-MDT0001 complete [ 8576.661036] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8579.498885] Key type lgssc unregistered [ 8579.828656] LNet: 266545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8579.841744] LNetError: 266545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8579.873624] LNet: Removed LNI 192.168.201.139@tcp [ 8580.656463] Key type .llcrypt unregistered [ 8580.658980] Key type ._llcrypt unregistered [ 8599.097866] Key type ._llcrypt registered [ 8599.103869] Key type .llcrypt registered [ 8599.212435] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8600.221649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8600.243410] alg: No test for adler32 (adler32-zlib) [ 8601.304718] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8601.554798] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8603.223208] Key type lgssc registered [ 8604.472105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8612.770594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8612.785407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.354275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.666162] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8624.610794] LustreError: 267768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8624.623246] LustreError: 267768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8627.664771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.983143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.640131] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8639.744898] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8645.382885] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8652.944493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8653.303312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.457318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8662.085810] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8671.749393] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8685.026647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8685.029856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.045681] LustreError: Skipped 1 previous similar message [ 8685.049041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8685.059722] Lustre: Skipped 1 previous similar message [ 8685.093326] Lustre: Skipped 1 previous similar message [ 8690.145438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8690.161846] Lustre: Skipped 1 previous similar message [ 8690.314307] Lustre: server umount lustre-OST0000 complete [ 8694.755660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8694.759159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.783483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8700.385935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8700.396578] Lustre: Skipped 3 previous similar messages [ 8700.615358] Lustre: server umount lustre-MDT0000 complete [ 8705.299338] LustreError: 267749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618029 with bad export cookie 11203338442639910549 [ 8705.310413] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.315513] LustreError: 267749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8705.832829] Lustre: server umount lustre-MDT0001 complete [ 8718.639349] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8722.048851] Key type lgssc unregistered [ 8722.341901] LNet: 271449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8722.346674] LNetError: 271449:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8722.361880] LNet: Removed LNI 192.168.201.139@tcp [ 8723.103351] Key type .llcrypt unregistered [ 8723.105709] Key type ._llcrypt unregistered [ 8750.031344] Key type ._llcrypt registered [ 8750.036515] Key type .llcrypt registered [ 8750.187514] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8751.599848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8751.813737] alg: No test for adler32 (adler32-zlib) [ 8752.916622] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8753.224872] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8754.959200] Key type lgssc registered [ 8756.173939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8763.745461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8763.766802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.119762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.448843] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8775.651218] LustreError: 272873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.660515] LustreError: 272873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8780.562649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.774590] LustreError: 272872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.277809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8786.877163] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8797.258379] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8804.676710] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8814.778828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8815.288300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8821.711712] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8829.044551] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8835.770533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8867.808819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8867.825458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8867.864789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8871.910860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8871.911238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.946209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8873.168529] Lustre: server umount lustre-OST0000 complete [ 8878.567321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8878.576742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.589566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8878.595440] Lustre: Skipped 1 previous similar message [ 8882.146232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8882.154359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.411788] Lustre: server umount lustre-MDT0000 complete [ 8887.270073] LustreError: 272872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8887.303588] LustreError: 272872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8888.877046] LustreError: 272854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618213 with bad export cookie 4520304052785797583 [ 8888.879578] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8888.895951] LustreError: 272854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8889.201735] Lustre: server umount lustre-MDT0001 complete [ 8902.064773] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 8905.542175] Key type lgssc unregistered [ 8905.914194] LNet: 277204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8905.917826] LNetError: 277204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8905.939662] LNet: Removed LNI 192.168.201.139@tcp [ 8907.016834] Key type .llcrypt unregistered [ 8907.019237] Key type ._llcrypt unregistered [ 8931.048742] Key type ._llcrypt registered [ 8931.059809] Key type .llcrypt registered [ 8931.223905] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 8933.562777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8933.614127] alg: No test for adler32 (adler32-zlib) [ 8934.850477] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 8935.019382] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 8936.631773] Key type lgssc registered [ 8937.493776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8945.226523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8945.235550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.578956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8950.801912] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8955.879542] LustreError: 278442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8955.894924] LustreError: 278442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8958.875470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.339580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8964.820962] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 8973.864615] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8980.524557] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8988.481682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8988.943607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8991.031840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8995.309198] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9003.592279] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9016.801930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9016.811813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.848811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.875805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9019.879532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.890787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9021.777712] Lustre: server umount lustre-OST0000 complete [ 9030.114331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9030.123767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.150800] Lustre: Skipped 1 previous similar message [ 9030.166406] Lustre: Skipped 2 previous similar messages [ 9035.233985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.354537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.364939] Lustre: Skipped 2 previous similar messages [ 9041.375646] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.671495] Lustre: server umount lustre-MDT0000 complete [ 9045.473729] LustreError: 278454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.511616] LustreError: 278454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9046.028125] LustreError: 278408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618370 with bad export cookie 6315314883441970942 [ 9046.045130] LustreError: 278408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9046.048705] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9046.647956] Lustre: server umount lustre-MDT0001 complete [ 9060.351942] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9063.372568] Key type lgssc unregistered [ 9063.657672] LNet: 282141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9063.665784] LNetError: 282141:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9063.687106] LNet: Removed LNI 192.168.201.139@tcp [ 9064.616991] Key type .llcrypt unregistered [ 9064.618975] Key type ._llcrypt unregistered [ 9081.246447] Key type ._llcrypt registered [ 9081.253596] Key type .llcrypt registered [ 9081.464852] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9101.026956] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9117.396505] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9118.497152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9118.604636] alg: No test for adler32 (adler32-zlib) [ 9119.718329] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 9120.055058] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9121.767602] Key type lgssc registered [ 9122.859980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9159.261784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9159.327740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.634534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9160.709992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9160.875350] Lustre: lustre-MDT0000: new disk, initializing [ 9161.016090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9161.034527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9165.891754] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9177.872852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.018299] Lustre: 285695:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.074410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9178.077457] Lustre: Skipped 1 previous similar message [ 9178.156844] Lustre: lustre-MDT0001: new disk, initializing [ 9178.253286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9178.289190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9178.303700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9183.222491] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9193.173789] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9200.688496] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9208.571434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.849900] Lustre: lustre-OST0000: new disk, initializing [ 9208.854927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9208.867744] Lustre: 287667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9208.980470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.084768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9215.125985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9215.261310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9216.800447] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9227.713588] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9232.614546] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9232.807201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9237.720991] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9237.892653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9241.071303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9241.071986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.089725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9241.094384] Lustre: Skipped 1 previous similar message [ 9246.181268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9246.191422] Lustre: Skipped 2 previous similar messages [ 9251.298274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9251.307735] Lustre: Skipped 1 previous similar message [ 9254.367159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9254.703149] Lustre: server umount lustre-OST0000 complete [ 9260.519850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9260.539574] LustreError: Skipped 1 previous similar message [ 9260.556321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.585475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.143426] Lustre: server umount lustre-MDT0000 complete [ 9270.572583] LustreError: 285686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618594 with bad export cookie 16598806512735488005 [ 9270.583276] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.584575] LustreError: 285686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9271.034798] Lustre: server umount lustre-MDT0001 complete [ 9282.631656] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:50:04 (1782618604) [ 9291.502190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.257192] LustreError: 290081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9292.278436] LustreError: 290081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9292.369414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9297.261827] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9297.380737] LustreError: 290082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.499900] LustreError: 290081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9307.620420] LustreError: 290082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9308.035683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.565254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9314.131626] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9322.851125] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9328.810555] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9335.408654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9336.007808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9342.352515] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9351.217958] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9369.056561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9369.070251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.096835] Lustre: Skipped 1 previous similar message [ 9369.109510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9369.112249] Lustre: Skipped 2 previous similar messages [ 9371.369091] Lustre: server umount lustre-OST0000 complete [ 9375.200080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.209848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9375.225904] Lustre: Skipped 1 previous similar message [ 9375.245376] LustreError: Skipped 1 previous similar message [ 9380.320649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9380.333906] Lustre: Skipped 5 previous similar messages [ 9381.643225] Lustre: server umount lustre-MDT0000 complete [ 9385.442644] LustreError: 291041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.462320] LustreError: 291041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9385.707232] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618709 with bad export cookie 16598806512735488887 [ 9385.724144] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.996500] Lustre: server umount lustre-MDT0001 complete [ 9398.815426] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9402.104687] Key type lgssc unregistered [ 9402.406514] LNet: 293641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9402.412874] LNetError: 293641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9402.427202] LNet: Removed LNI 192.168.201.139@tcp [ 9403.293945] Key type .llcrypt unregistered [ 9403.296340] Key type ._llcrypt unregistered [ 9431.486460] Key type ._llcrypt registered [ 9431.488481] Key type .llcrypt registered [ 9431.625238] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9446.199777] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9447.265363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9447.281363] alg: No test for adler32 (adler32-zlib) [ 9448.415601] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 9448.778143] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [ 9450.471138] Key type lgssc registered [ 9451.764394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9499.696224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9499.721054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.041405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9501.081090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9501.210293] Lustre: lustre-MDT0000: new disk, initializing [ 9501.353372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9501.385380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9505.988864] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9518.302444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.555681] Lustre: 297464:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9518.610480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9518.614021] Lustre: Skipped 1 previous similar message [ 9518.698555] Lustre: lustre-MDT0001: new disk, initializing [ 9518.850305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9518.904305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9518.928925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9523.993661] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9534.546494] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9541.036323] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9549.631453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9549.957943] Lustre: lustre-OST2710: new disk, initializing [ 9549.962553] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9549.970766] Lustre: 299434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9550.093447] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9555.519310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9555.542995] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9555.687057] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9557.441940] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9569.512591] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9577.248337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.503907] Lustre: lustre-OST03e8: new disk, initializing [ 9577.507530] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9577.514700] Lustre: 300498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9577.588937] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9584.916360] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9585.861368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9585.878124] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9586.019392] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9593.266736] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9601.102245] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9601.322207] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9607.000196] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9607.296203] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9613.082971] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9613.513497] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9618.488322] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9618.695682] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9624.624958] Lustre: server umount lustre-MDT0000 complete [ 9626.592872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9626.599684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9627.125303] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.674763] LustreError: 297458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782618952 with bad export cookie 2819915335148709121 [ 9628.681856] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9628.688765] LustreError: 297458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9629.344382] Lustre: server umount lustre-MDT0001 complete [ 9643.077640] Lustre: server umount lustre-OST2710 complete [ 9657.943830] Lustre: server umount lustre-OST03e8 complete [ 9679.489185] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [ 9692.182985] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9737.199530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.484795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9737.512063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9737.622873] Lustre: lustre-MDT0000: new disk, initializing [ 9737.678627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9737.693393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9744.268261] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9756.341258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.493304] Lustre: 305752:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9756.573943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9756.585788] Lustre: Skipped 1 previous similar message [ 9756.698337] Lustre: lustre-MDT0001: new disk, initializing [ 9756.819472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9756.889948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9756.911895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9762.526779] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9772.122577] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9777.530990] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9784.765212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.123536] Lustre: lustre-OST0000: new disk, initializing [ 9785.127548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9785.133562] Lustre: 307721:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9785.229978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9790.967932] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9793.072523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9793.088532] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9793.228717] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9800.071429] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9805.185526] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9805.475179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9810.548653] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9810.814252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9813.985911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9813.991813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.998180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9813.998218] LustreError: Skipped 1 previous similar message [ 9814.034930] Lustre: Skipped 3 previous similar messages [ 9818.963124] Lustre: server umount lustre-OST0000 complete [ 9823.712884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9823.721904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.738980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9823.754504] Lustre: Skipped 1 previous similar message [ 9829.346535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9829.364254] Lustre: Skipped 3 previous similar messages [ 9829.490808] Lustre: server umount lustre-MDT0000 complete [ 9834.070586] LustreError: 305745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619158 with bad export cookie 2819915335148710808 [ 9834.085415] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9834.086373] LustreError: 305745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9834.563886] Lustre: server umount lustre-MDT0001 complete [ 9846.145161] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:59:28 (1782619168) [ 9848.209457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9850.298278] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:59:33 (1782619173) [ 9854.731483] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9889.426336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.902985] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9889.963818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9890.257186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9896.179494] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9906.083040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9906.141596] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9906.164036] Lustre: 311583:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9911.805533] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [ 9934.965651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9935.130461] Lustre: Denying initial registration attempt from nid 192.168.201.139@tcp, specified as failover [ 9935.143451] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9935.157496] LustreError: 313036:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9935.166202] LustreError: 313036:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9935.175762] LustreError: 313036:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9935.251537] Lustre: server umount lustre-OST0000 complete [ 9935.255273] LustreError: 313036:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9944.033670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9944.047287] Lustre: Skipped 1 previous similar message [ 9944.068079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9949.152517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9949.161209] Lustre: Skipped 2 previous similar messages [ 9957.343175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9957.569566] Lustre: server umount lustre-MDT0000 complete [ 9959.397731] LustreError: 311593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.436035] LustreError: 311593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9962.822888] LustreError: 311575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619286 with bad export cookie 2819915335148711690 [ 9962.843867] LustreError: 311575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9962.846110] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9963.749763] Lustre: server umount lustre-MDT0001 complete [ 9976.418247] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [ 9980.240345] Key type lgssc unregistered [ 9980.683971] LNet: 314171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9980.713861] LNetError: 314171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9981.743373] LNet: Removed LNI 192.168.201.139@tcp [ 9982.869898] Key type .llcrypt unregistered [ 9982.872218] Key type ._llcrypt unregistered [ 9996.851927] Key type ._llcrypt registered [ 9996.857734] Key type .llcrypt registered [ 9997.065892] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [ 9999.139772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9999.193937] alg: No test for adler32 (adler32-zlib) [10000.535653] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [10000.869391] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10002.608843] Key type lgssc registered [10003.403382] Lustre: Echo OBD driver; http://www.lustre.org/ [10058.662319] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10067.456552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10067.474608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10068.749678] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10068.796930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10069.035642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10073.660191] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10083.590449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10083.758914] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10083.824854] Lustre: 316939:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10084.201854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10088.793416] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10097.581543] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10104.972981] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10127.464477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.720399] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10127.991135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10134.611132] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10143.425199] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10146.286559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10146.289500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.302219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10146.321556] Lustre: Skipped 1 previous similar message [10146.341161] Lustre: Skipped 1 previous similar message [10151.406984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10151.417321] Lustre: Skipped 1 previous similar message [10152.246264] Lustre: server umount lustre-OST0000 complete [10161.120938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10161.132262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10161.145582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.764254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10166.782042] Lustre: Skipped 2 previous similar messages [10171.872442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.881197] Lustre: Skipped 1 previous similar message [10172.385422] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10172.686736] Lustre: server umount lustre-MDT0000 complete [10177.049266] LustreError: 316931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619501 with bad export cookie 9680379306600161308 [10177.055939] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.059080] LustreError: 316931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10177.547671] Lustre: server umount lustre-MDT0001 complete [10189.723342] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10192.882220] Key type lgssc unregistered [10193.159656] LNet: 320516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10193.193479] LNetError: 320516:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10194.213711] LNet: Removed LNI 192.168.201.139@tcp [10195.295783] Key type .llcrypt unregistered [10195.297524] Key type ._llcrypt unregistered [10224.287491] Key type ._llcrypt registered [10224.289878] Key type .llcrypt registered [10224.431876] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10226.087778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10226.161585] alg: No test for adler32 (adler32-zlib) [10227.402511] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [10227.832293] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10229.759272] Key type lgssc registered [10231.410681] Lustre: Echo OBD driver; http://www.lustre.org/ [10240.422596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10240.444993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.006657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10246.714125] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10252.263331] LustreError: 321935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.287147] LustreError: 321935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10256.113757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10256.660898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10262.221032] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10270.922435] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10277.721691] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10287.741501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.161919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10294.710211] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10301.939351] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10322.914714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.925256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10322.937548] Lustre: Skipped 1 previous similar message [10324.007883] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [10324.024936] Lustre: Skipped 2 previous similar messages [10326.927446] Lustre: server umount lustre-MDT0000 complete [10328.047947] LustreError: 321935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10328.077063] LustreError: 321935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10329.089737] LustreError: 321932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.964482] LustreError: 321917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619655 with bad export cookie 12333800487915344526 [10330.974078] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10330.978465] LustreError: 321917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10333.152154] LustreError: 321930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10333.155193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.195155] Lustre: Skipped 1 previous similar message [10333.213267] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10337.473095] Lustre: server umount lustre-MDT0001 complete [10344.856196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10353.987815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.192193] LustreError: 325538:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10356.200565] LustreError: 325538:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000005dbbb681 x1869212078354432/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1782619680 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10356.233348] LustreError: 325538:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10356.639962] LustreError: 321244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007da5e3ec x1869212078354560/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10357.017599] LustreError: 323768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.034580] LustreError: 323768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10357.096859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10357.161637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10362.025220] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10365.289849] LustreError: 325988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10365.294764] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10365.302587] LustreError: 325988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10371.188896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.647948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10376.684687] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10376.699529] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10384.868324] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10386.752030] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10392.858767] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10395.268310] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10402.272190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10402.288625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.295045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10402.310431] Lustre: Skipped 1 previous similar message [10402.323219] Lustre: Skipped 3 previous similar messages [10407.404764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10407.412625] Lustre: Skipped 1 previous similar message [10407.859232] Lustre: server umount lustre-OST0000 complete [10412.512406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10412.517186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.544526] Lustre: Skipped 1 previous similar message [10417.638933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10417.648422] Lustre: Skipped 3 previous similar messages [10418.952364] Lustre: server umount lustre-MDT0000 complete [10422.757039] LustreError: 325556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.783279] LustreError: 325556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10423.347501] LustreError: 321916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619747 with bad export cookie 12333800487915352163 [10423.357195] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10423.364665] LustreError: 321916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10423.865567] Lustre: server umount lustre-MDT0001 complete [10435.744449] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10438.432693] Key type lgssc unregistered [10438.703733] LNet: 328227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10438.714078] LNetError: 328227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10438.732620] LNet: Removed LNI 192.168.201.139@tcp [10439.707351] Key type .llcrypt unregistered [10439.711879] Key type ._llcrypt unregistered [10467.736958] Key type ._llcrypt registered [10467.738801] Key type .llcrypt registered [10467.834126] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10468.971409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10469.032058] alg: No test for adler32 (adler32-zlib) [10470.216387] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [10470.478436] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10472.167289] Key type lgssc registered [10473.530403] Lustre: Echo OBD driver; http://www.lustre.org/ [10480.923609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10480.935615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.292350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10486.981870] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10491.875302] LustreError: 329646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.895864] LustreError: 329646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10496.766864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10497.002723] LustreError: 329645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10497.306990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10501.714704] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10509.841441] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10516.455775] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10519.783157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.788058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10519.789817] Lustre: Skipped 1 previous similar message [10519.798426] Lustre: Skipped 1 previous similar message [10522.850584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10522.854164] Lustre: Skipped 1 previous similar message [10528.225137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.231961] Lustre: Skipped 1 previous similar message [10533.344853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10533.356320] Lustre: Skipped 1 previous similar message [10534.373047] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10534.622153] Lustre: server umount lustre-MDT0000 complete [10538.464449] LustreError: 329645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10538.486771] LustreError: 329645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10538.652238] LustreError: 329625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782619862 with bad export cookie 6348432322910313717 [10538.659436] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10538.660788] LustreError: 329625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10539.130396] Lustre: server umount lustre-MDT0001 complete [10556.989704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10557.273550] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10557.300911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10557.504775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10563.506330] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10573.097397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.172460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10573.199613] Lustre: 332235:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10573.516199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10579.166861] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10587.197148] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10593.312480] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10603.308899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10603.492690] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10603.514376] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10603.527777] LustreError: 334121:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10603.536481] LustreError: 334121:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10603.542228] LustreError: 334121:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10603.618988] Lustre: server umount lustre-OST0000 complete [10603.624485] LustreError: 334121:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -2 [10614.950924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10615.228550] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10615.466802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10621.110170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10621.298216] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10628.654728] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10637.081551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10637.165542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10637.300373] Lustre: lustre-OST0001: new disk, initializing [10637.306344] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10637.316431] Lustre: 335660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10637.435690] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10643.597435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10643.617779] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10643.741060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10644.019489] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10652.075298] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10659.296065] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10659.300684] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.308067] LustreError: Skipped 1 previous similar message [10659.311569] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10659.330704] Lustre: Skipped 1 previous similar message [10659.361518] Lustre: Skipped 1 previous similar message [10662.342522] Lustre: server umount lustre-OST0001 complete [10664.433473] LustreError: 334587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.464905] LustreError: 334587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10666.983965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10666.992394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10667.495414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10667.503117] Lustre: Skipped 1 previous similar message [10669.539899] LustreError: 335952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.560916] LustreError: 335952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10672.920102] Lustre: server umount lustre-OST0000 complete [10679.797630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.835432] Lustre: Skipped 2 previous similar messages [10684.843517] Lustre: server umount lustre-MDT0000 complete [10684.899250] LustreError: 332243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.927651] LustreError: 332243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10689.298244] LustreError: 332229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620013 with bad export cookie 6348432322910314270 [10689.328176] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10689.334517] LustreError: 332229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10689.920460] Lustre: server umount lustre-MDT0001 complete [10701.456215] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [10704.085180] Key type lgssc unregistered [10704.433353] LNet: 337621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10704.439536] LNetError: 337621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10704.461740] LNet: Removed LNI 192.168.201.139@tcp [10705.311214] Key type .llcrypt unregistered [10705.312780] Key type ._llcrypt unregistered [10776.833474] Key type ._llcrypt registered [10776.835644] Key type .llcrypt registered [10776.972289] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [10792.848091] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [10793.711524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10793.828175] alg: No test for adler32 (adler32-zlib) [10794.842552] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [10795.068182] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [10796.751194] Key type lgssc registered [10798.064804] Lustre: Echo OBD driver; http://www.lustre.org/ [10838.046313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10838.088378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.406979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10839.470304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10839.560735] Lustre: lustre-MDT0000: new disk, initializing [10839.627438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.643805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10844.831905] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10856.213175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.393748] Lustre: 342554:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10856.468865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10856.480633] Lustre: Skipped 1 previous similar message [10856.576957] Lustre: lustre-MDT0001: new disk, initializing [10856.694615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10856.749838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10856.767954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10861.365931] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10871.751464] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10878.286418] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10886.724261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.180712] Lustre: lustre-OST0000: new disk, initializing [10887.183856] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10887.189637] Lustre: 344522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10887.262287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.609252] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10894.980390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10894.998767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10895.191860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10904.633079] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10910.016834] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10910.222090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10915.315659] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10915.599122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10920.929902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.939083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10920.950211] Lustre: Skipped 1 previous similar message [10920.973847] Lustre: Skipped 1 previous similar message [10923.860646] Lustre: server umount lustre-OST0000 complete [10928.608500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10928.615151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10928.632454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10931.185077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10931.190210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10934.541259] Lustre: server umount lustre-MDT0000 complete [10938.760954] LustreError: 342548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620262 with bad export cookie 6753352776917006758 [10938.771740] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10938.781650] LustreError: 342548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10939.183387] Lustre: server umount lustre-MDT0001 complete [10948.917608] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:17:51 (1782620271) [10957.659741] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:18:00 (1782620280) [10967.147185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.544036] LustreError: 347279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10967.563125] LustreError: 347279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10967.633344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10972.199047] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10972.643323] LustreError: 347280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10977.762312] LustreError: 347279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10980.590135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10980.996214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10985.348269] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [10992.764775] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10998.233560] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11006.084892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11006.377871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11012.722598] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11019.599214] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11026.176604] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11028.069623] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11029.734733] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11031.369690] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11032.993648] Lustre: Failing over lustre-MDT0000 [11033.249421] Lustre: server umount lustre-MDT0000 complete [11034.086450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11034.098156] Lustre: Skipped 2 previous similar messages [11034.099010] LustreError: 347275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.124377] LustreError: 347275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11039.202382] LustreError: 348008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.218581] LustreError: 348008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11049.444408] LustreError: 347279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.459411] LustreError: 347279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11050.463572] Lustre: 340779:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782620358/real 1782620358] req@000000006e374912 x1869212673973248/t0(0) o400->MGC192.168.201.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782620374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11050.489170] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11052.063476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.988354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11061.040207] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11063.304790] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11065.489338] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11066.355053] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11066.417477] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11066.467164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11074.036695] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11075.758528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11079.120824] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11081.000090] Lustre: Failing over lustre-MDT0000 [11081.293924] Lustre: server umount lustre-MDT0000 complete [11081.695835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11081.700615] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11081.701850] LustreError: 347275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.701858] LustreError: 347275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [11081.739584] Lustre: Skipped 2 previous similar messages [11100.121287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11100.343693] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.614926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11100.666647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11104.594765] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11105.293410] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11105.776698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11105.783516] Lustre: Skipped 2 previous similar messages [11105.814701] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11105.861657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11112.869759] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11114.550617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11121.120709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11121.122616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11121.129031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.129042] Lustre: Skipped 1 previous similar message [11121.148163] Lustre: Skipped 2 previous similar messages [11126.015626] Lustre: server umount lustre-MDT0000 complete [11126.243245] LustreError: 350801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.264246] LustreError: 350801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [11129.415091] LustreError: 347261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620453 with bad export cookie 6753352776917009467 [11129.422983] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11129.425751] LustreError: 347261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11129.731576] Lustre: server umount lustre-MDT0001 complete [11143.834009] Lustre: server umount lustre-OST0000 complete [11156.328979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.156334] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11166.975663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11167.247650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11167.250570] Lustre: Skipped 1 previous similar message [11171.018432] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11177.818855] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11182.432440] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11187.996697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11194.122427] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11201.485605] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11201.596370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11206.627602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.632149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11206.640926] Lustre: Skipped 1 previous similar message [11206.657055] Lustre: Skipped 3 previous similar messages [11211.680478] Lustre: server umount lustre-OST0000 complete [11216.865367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.337218] Lustre: server umount lustre-MDT0000 complete [11221.984458] LustreError: 353617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11221.994761] LustreError: 353617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11224.687298] LustreError: 353598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620548 with bad export cookie 6753352776917010216 [11224.688099] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11224.696069] LustreError: 353598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11224.948958] Lustre: server umount lustre-MDT0001 complete [11233.374230] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11236.170484] Key type lgssc unregistered [11236.449960] LNet: 357148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11236.454459] LNetError: 357148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11236.476231] LNet: Removed LNI 192.168.201.139@tcp [11237.497913] Key type .llcrypt unregistered [11237.499925] Key type ._llcrypt unregistered [11260.109496] Key type ._llcrypt registered [11260.113993] Key type .llcrypt registered [11260.223323] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11261.187739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11261.203815] alg: No test for adler32 (adler32-zlib) [11262.236348] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [11262.387380] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11264.016050] Key type lgssc registered [11264.710171] Lustre: Echo OBD driver; http://www.lustre.org/ [11269.415677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11269.431875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11270.696443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11273.534736] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11280.054636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11280.303726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11284.161835] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11290.352427] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11294.902070] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11299.890599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11300.165597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11305.478943] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11311.891941] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11316.533115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11326.432242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11326.438143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.458527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.945660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11326.957329] Lustre: Skipped 1 previous similar message [11326.963164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11326.966417] Lustre: Skipped 1 previous similar message [11331.597379] Lustre: server umount lustre-MDT0000 complete [11332.072719] LustreError: 358649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11332.090885] LustreError: 358649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11334.248398] LustreError: 358630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620658 with bad export cookie 11613528832232130054 [11334.252569] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.261150] LustreError: 358630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11334.512699] Lustre: server umount lustre-MDT0001 complete [11347.982945] Lustre: server umount lustre-OST0000 complete [11357.539411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.886379] LustreError: 362392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11357.940936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11361.056974] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11363.298818] LustreError: 362393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.172167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.480495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11370.764393] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11376.290139] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11380.695395] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11385.594805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11385.914389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11391.397194] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11396.991738] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11403.314906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11418.597536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11418.602573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11418.602712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11418.614906] Lustre: Skipped 1 previous similar message [11418.631799] Lustre: Skipped 2 previous similar messages [11423.714335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11423.736481] Lustre: Skipped 2 previous similar messages [11424.236608] Lustre: server umount lustre-MDT0000 complete [11427.036197] LustreError: 363125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620751 with bad export cookie 11613528832232131160 [11427.041598] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11427.043025] LustreError: 363125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11427.272676] Lustre: server umount lustre-MDT0001 complete [11440.477820] Lustre: server umount lustre-OST0000 complete [11453.624275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11454.025846] LustreError: 366322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11454.136768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11457.772424] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11459.555319] LustreError: 366323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11464.220964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11467.984043] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11473.962372] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11478.075619] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11483.942066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11489.458523] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11495.102624] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11500.734850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11510.239913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11510.247248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11510.259234] Lustre: Skipped 1 previous similar message [11510.262118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11515.279337] Lustre: server umount lustre-OST0000 complete [11520.992969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11521.000550] LustreError: Skipped 1 previous similar message [11521.004044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11521.013075] Lustre: Skipped 1 previous similar message [11521.016181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11521.019292] Lustre: Skipped 2 previous similar messages [11532.255323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11532.412964] Lustre: server umount lustre-MDT0000 complete [11535.767470] LustreError: 366302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782620859 with bad export cookie 11613528832232132182 [11535.772500] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11535.773248] LustreError: 366302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11536.136623] Lustre: server umount lustre-MDT0001 complete [11544.491644] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11546.563729] Key type lgssc unregistered [11546.777965] LNet: 369872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11546.788390] LNetError: 369872:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11546.800258] LNet: Removed LNI 192.168.201.139@tcp [11547.492084] Key type .llcrypt unregistered [11547.494601] Key type ._llcrypt unregistered [11566.091405] Key type ._llcrypt registered [11566.099585] Key type .llcrypt registered [11566.184489] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11567.032473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11567.054615] alg: No test for adler32 (adler32-zlib) [11568.065610] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [11568.269784] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11569.897724] Key type lgssc registered [11570.711778] Lustre: Echo OBD driver; http://www.lustre.org/ [11576.307394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11576.322453] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.326924] LustreError: 371292:0:(osd_handler.c:8552:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11576.340945] LustreError: 371292:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11576.345319] LustreError: 371292:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11576.353075] LustreError: 371292:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11576.357120] LustreError: 371292:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -22 [11582.161434] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11582.173103] LustreError: 371690:0:(osd_handler.c:8552:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11582.203801] LustreError: 371690:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11582.211738] LustreError: 371690:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11582.218150] LustreError: 371690:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11582.225323] LustreError: 371690:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -22 [11595.432455] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11597.381760] Key type lgssc unregistered [11597.599218] LNet: 372692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11597.604255] LNetError: 372692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11597.615565] LNet: Removed LNI 192.168.201.139@tcp [11598.141502] Key type .llcrypt unregistered [11598.143045] Key type ._llcrypt unregistered [11611.749460] Key type ._llcrypt registered [11611.751301] Key type .llcrypt registered [11611.835888] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [11621.858376] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11622.357693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11622.373665] alg: No test for adler32 (adler32-zlib) [11623.399312] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [11623.574871] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11625.207197] Key type lgssc registered [11625.949941] Lustre: Echo OBD driver; http://www.lustre.org/ [11651.948285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11651.984266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.210909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11653.231840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11653.297463] Lustre: lustre-MDT0000: new disk, initializing [11653.374030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.392059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11657.322903] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11665.219510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.291996] Lustre: 375830:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11665.334931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11665.338294] Lustre: Skipped 1 previous similar message [11665.387151] Lustre: lustre-MDT0001: new disk, initializing [11665.452567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11665.483080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11665.488962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11668.058317] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11674.621855] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11678.007158] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11682.630736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11682.876311] Lustre: lustre-OST0000: new disk, initializing [11682.879790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11682.885568] Lustre: 377794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11682.967606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11687.752563] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11691.611466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11691.621127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11691.668082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11694.969303] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11698.460980] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11698.584132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11702.040988] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11702.211670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11707.365291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11707.366491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11707.373629] LustreError: Skipped 1 previous similar message [11707.376064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11707.395579] Lustre: Skipped 1 previous similar message [11712.485924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11712.488695] Lustre: Skipped 2 previous similar messages [11717.604578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11717.614393] Lustre: Skipped 1 previous similar message [11718.111634] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11718.200488] Lustre: server umount lustre-OST0000 complete [11721.697715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11721.709279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.738161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11722.721571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.153121] Lustre: server umount lustre-MDT0000 complete [11730.306297] LustreError: 375822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621054 with bad export cookie 11031834827122578078 [11730.322074] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11730.805689] Lustre: server umount lustre-MDT0001 complete [11738.509978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11739.719309] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:31:03 (1782621063) [11744.484914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.876439] LustreError: 380254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.897525] LustreError: 380254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11744.937784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11748.393835] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11750.369855] LustreError: 380255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11754.781264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.054167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11758.329171] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11764.236398] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11768.586246] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11773.574333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.802192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11778.192254] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11783.599591] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11789.084728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.217502] Lustre: lustre-OST0001: new disk, initializing [11789.219589] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11789.226062] Lustre: 383089:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11793.938147] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11796.002874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11796.009631] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11796.107073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11800.735640] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11806.184119] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11806.185241] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11806.194076] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11806.197444] Lustre: Skipped 3 previous similar messages [11809.775564] Lustre: server umount lustre-OST0001 complete [11811.300659] LustreError: 382074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.320541] LustreError: 382074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11816.422692] LustreError: 383257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.434989] LustreError: 383257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11821.544195] LustreError: 382077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11821.565657] LustreError: 382077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11826.144337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11826.154283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11826.168184] Lustre: Skipped 1 previous similar message [11826.175189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11826.187285] Lustre: Skipped 1 previous similar message [11830.738887] Lustre: server umount lustre-OST0000 complete [11836.896358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11836.899546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.924167] Lustre: Skipped 2 previous similar messages [11846.115338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11846.122265] Lustre: Skipped 6 previous similar messages [11848.164746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11848.336229] Lustre: server umount lustre-MDT0000 complete [11851.235702] LustreError: 380255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.261074] LustreError: 380255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11851.920067] LustreError: 380236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621176 with bad export cookie 11031834827122578960 [11851.923371] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11851.929975] LustreError: 380236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11858.504366] Lustre: server umount lustre-MDT0001 complete [11869.107076] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [11871.783373] Key type lgssc unregistered [11872.048124] LNet: 385073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11872.052746] LNetError: 385073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11872.075624] LNet: Removed LNI 192.168.201.139@tcp [11872.671146] Key type .llcrypt unregistered [11872.673128] Key type ._llcrypt unregistered [11914.585959] Key type ._llcrypt registered [11914.587840] Key type .llcrypt registered [11914.690830] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [11915.798578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11915.816976] alg: No test for adler32 (adler32-zlib) [11916.899379] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [11917.119764] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [11918.751191] Key type lgssc registered [11920.011709] Lustre: Echo OBD driver; http://www.lustre.org/ [11926.813841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11926.828948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.045341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11928.069858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11928.184704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11931.719662] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11938.533591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11938.601489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11938.627826] Lustre: 387852:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11938.882912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11941.789525] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11947.856340] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11952.499688] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11959.270578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11959.272681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.291322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11959.299977] Lustre: Skipped 1 previous similar message [11962.615047] Lustre: server umount lustre-MDT0000 complete [11965.709926] LustreError: 387845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621289 with bad export cookie 5079495149480365394 [11965.714592] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11965.728881] LustreError: 387845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11965.981371] Lustre: server umount lustre-MDT0001 complete [11973.490924] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:34:57 (1782621297) [11978.094847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11978.366616] LustreError: 390362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11978.379700] LustreError: 390362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11978.401427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11981.130360] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11983.846873] LustreError: 390363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.974468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.185102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11989.016742] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [11993.853466] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11997.249104] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12001.539299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12001.662336] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12001.814952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12006.194962] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12010.582984] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12015.584746] Lustre: Permanently deactivating lustre-OST0000 [12015.594619] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12021.765178] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12023.140171] LustreError: 393344:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12023.149039] LustreError: 393344:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [12023.153665] LustreError: 393344:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12024.758144] LustreError: 393393:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [12024.764855] LustreError: 393393:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12032.056396] Lustre: server umount lustre-OST0000 complete [12033.747834] LustreError: 393642:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [12033.755580] LustreError: 393642:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12037.602030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12037.605510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12037.616287] Lustre: Skipped 1 previous similar message [12037.625495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12039.139037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12039.147447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.256430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12044.260809] Lustre: Skipped 2 previous similar messages [12049.375201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12049.377068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12049.384725] Lustre: Skipped 1 previous similar message [12049.466052] Lustre: server umount lustre-MDT0000 complete [12051.619824] LustreError: 390344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621375 with bad export cookie 5079495149480365975 [12051.620659] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12051.625055] LustreError: 390344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12061.241591] Lustre: server umount lustre-MDT0001 complete [12066.579950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12069.423987] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12071.345773] LustreError: 394803:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12075.641157] Lustre: 395142:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.139@tcp [12077.125717] Lustre: server umount MGS complete [12084.744287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12085.015764] LustreError: 395865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.037058] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12085.042058] Lustre: Skipped 1 previous similar message [12085.086600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12088.191895] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12090.337976] LustreError: 395866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.214741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.429826] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12097.306798] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12101.931741] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12105.459661] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12106.810971] Lustre: Permanently reactivating lustre-OST0000 [12106.822978] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12106.828654] Lustre: Skipped 2 previous similar messages [12133.855209] LustreError: 397646:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12138.923079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.140247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12139.144863] Lustre: Skipped 1 previous similar message [12142.816349] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12147.816990] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12158.946437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12158.952708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.967192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12161.722301] Lustre: server umount lustre-OST0000 complete [12165.092043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12165.104252] Lustre: Skipped 2 previous similar messages [12170.499664] Lustre: server umount lustre-MDT0000 complete [12172.807486] LustreError: 395846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621496 with bad export cookie 5079495149480367025 [12172.811278] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.812825] LustreError: 395846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12173.020167] Lustre: server umount lustre-MDT0001 complete [12181.410925] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12183.348724] Key type lgssc unregistered [12183.519491] LNet: 400374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12183.527949] LNetError: 400374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12184.554178] LNet: Removed LNI 192.168.201.139@tcp [12185.131831] Key type .llcrypt unregistered [12185.132775] Key type ._llcrypt unregistered [12195.760602] Key type ._llcrypt registered [12195.762905] Key type .llcrypt registered [12195.842348] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [12204.604247] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12205.098141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12205.153121] alg: No test for adler32 (adler32-zlib) [12206.045446] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12206.181095] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12207.783939] Key type lgssc registered [12208.419735] Lustre: Echo OBD driver; http://www.lustre.org/ [12227.143660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12227.155443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12228.320391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12228.340536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12228.398274] Lustre: lustre-MDT0000: new disk, initializing [12228.446415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.460523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12230.836858] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12237.177929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12237.232196] Lustre: 403507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12237.270752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12237.273812] Lustre: Skipped 1 previous similar message [12237.324359] Lustre: lustre-MDT0001: new disk, initializing [12237.390056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12237.405917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12237.410873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12239.629279] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12244.983207] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12247.871930] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12251.734994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12251.907300] Lustre: lustre-OST0000: new disk, initializing [12251.909570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12251.914449] Lustre: 405472:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12251.960984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12255.160895] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12259.867923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12259.876579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12259.922446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12260.973275] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12263.300362] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12263.476654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12265.666607] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12265.805478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12270.052312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12270.053176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.054657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12270.056123] LustreError: Skipped 1 previous similar message [12270.067825] Lustre: Skipped 1 previous similar message [12275.172555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12275.175023] Lustre: Skipped 2 previous similar messages [12280.291899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12280.297042] Lustre: Skipped 1 previous similar message [12281.314657] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12281.405615] Lustre: server umount lustre-OST0000 complete [12285.410428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12285.415673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12285.418021] Lustre: Skipped 1 previous similar message [12285.424573] Lustre: Skipped 1 previous similar message [12289.592125] Lustre: server umount lustre-MDT0000 complete [12291.726499] LustreError: 404301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621615 with bad export cookie 2223559039209941256 [12291.727507] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12291.735139] LustreError: 404301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12291.959765] Lustre: server umount lustre-MDT0001 complete [12297.448666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12298.438699] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:40:22 (1782621622) [12302.350563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.636595] LustreError: 407930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.646673] LustreError: 407930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12302.681303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12305.022840] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12307.935906] LustreError: 407931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12309.518235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12309.742613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12311.986695] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12316.184943] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12319.079182] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12322.653680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12322.827590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12325.789687] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12329.647973] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12330.675712] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12332.542114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12342.243836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12342.248504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.255296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12344.290084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12344.293127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12354.015995] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12354.125484] Lustre: server umount lustre-OST0000 complete [12359.649487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.650605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12359.657779] Lustre: Skipped 1 previous similar message [12359.662070] Lustre: Skipped 5 previous similar messages [12362.239081] Lustre: server umount lustre-MDT0000 complete [12364.196232] LustreError: 407910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621688 with bad export cookie 2223559039209942138 [12364.199100] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12364.200754] LustreError: 407910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12364.388247] Lustre: server umount lustre-MDT0001 complete [12369.797422] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12371.253974] Key type lgssc unregistered [12371.421689] LNet: 411652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12371.425213] LNetError: 411652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12371.433574] LNet: Removed LNI 192.168.201.139@tcp [12371.835639] Key type .llcrypt unregistered [12371.837600] Key type ._llcrypt unregistered [12383.731347] Key type ._llcrypt registered [12383.733214] Key type .llcrypt registered [12383.785703] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12383.931446] Key type .llcrypt unregistered [12383.934652] Key type ._llcrypt unregistered [12393.627482] Key type ._llcrypt registered [12393.628952] Key type .llcrypt registered [12393.675624] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12394.147275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12394.155328] alg: No test for adler32 (adler32-zlib) [12395.058350] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12395.158166] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12396.751245] Key type lgssc registered [12397.328769] Lustre: Echo OBD driver; http://www.lustre.org/ [12401.501588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12401.516537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12402.777276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12404.855216] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12408.953812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12409.178734] LustreError: 413852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12409.190967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12409.193934] LustreError: 413852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12412.257789] LustreError: 413853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.266881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12412.278463] LustreError: 413853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12412.344763] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12415.778513] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12417.506195] LustreError: 413849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.512016] LustreError: 413849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12419.076918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12419.325974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12421.596913] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12432.868133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12432.873405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12432.882813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12434.917979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12434.918348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12434.921418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.799236] Lustre: server umount lustre-OST0000 complete [12440.033090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12440.033777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12440.034534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12440.034541] Lustre: Skipped 1 previous similar message [12440.039147] Lustre: Skipped 1 previous similar message [12445.152715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12445.157308] Lustre: Skipped 2 previous similar messages [12450.275920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12450.280107] Lustre: Skipped 1 previous similar message [12453.343273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12453.430791] Lustre: server umount lustre-MDT0000 complete [12455.396051] LustreError: 413852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12455.408916] LustreError: 413852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12455.530973] LustreError: 415592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621779 with bad export cookie 1482181017978039193 [12455.533462] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12455.536574] LustreError: 415592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12455.766952] Lustre: server umount lustre-MDT0001 complete [12461.126619] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12462.627535] Key type lgssc unregistered [12462.777167] LNet: 416997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12462.784664] LNetError: 416997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12462.804627] LNet: Removed LNI 192.168.201.139@tcp [12463.300486] Key type .llcrypt unregistered [12463.302370] Key type ._llcrypt unregistered [12476.942111] Key type ._llcrypt registered [12476.943669] Key type .llcrypt registered [12477.004372] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12477.580299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12477.596521] alg: No test for adler32 (adler32-zlib) [12478.535191] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12478.675967] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12480.287185] Key type lgssc registered [12480.859653] Lustre: Echo OBD driver; http://www.lustre.org/ [12484.856721] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12484.870298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12500.319453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12505.439295] LustreError: 418365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.139@tcp: failed processing log, type 4: rc = -110 [12532.063246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12535.784479] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12539.447914] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12542.751223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12542.986982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12543.032612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12545.356842] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12550.277214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.517535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12552.534629] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12556.150640] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12559.047076] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12563.425630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12563.429317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.437906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.993530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.994616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.999411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12566.002736] Lustre: Skipped 1 previous similar message [12569.119675] Lustre: server umount lustre-OST0000 complete [12571.104989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.106380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.106942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12571.111486] Lustre: Skipped 1 previous similar message [12576.224183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12576.226552] Lustre: Skipped 2 previous similar messages [12576.990737] Lustre: server umount lustre-MDT0000 complete [12578.581333] LustreError: 418372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621902 with bad export cookie 1811516760114072648 [12578.581643] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12578.593177] LustreError: 418372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12578.767329] Lustre: server umount lustre-MDT0001 complete [12583.123334] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12584.545431] Key type lgssc unregistered [12584.708402] LNet: 421931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12584.714266] LNetError: 421931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12584.727049] LNet: Removed LNI 192.168.201.139@tcp [12585.173280] Key type .llcrypt unregistered [12585.175624] Key type ._llcrypt unregistered [12596.079435] Key type ._llcrypt registered [12596.081183] Key type .llcrypt registered [12596.135129] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12596.498402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12596.549107] alg: No test for adler32 (adler32-zlib) [12597.428392] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12597.532537] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12599.119171] Key type lgssc registered [12599.675221] Lustre: Echo OBD driver; http://www.lustre.org/ [12603.203339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12603.211779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.419952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12606.107846] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12609.792621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.981442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12611.690635] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12614.643974] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12617.014911] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12619.813058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12619.940408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12622.073924] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12624.971605] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12629.160549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12634.596543] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12634.599725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12634.606042] Lustre: Skipped 2 previous similar messages [12634.613732] Lustre: Skipped 2 previous similar messages [12636.683613] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.39@tcp (stopping) [12638.633857] Lustre: server umount lustre-MDT0000 complete [12639.676320] LustreError: 423307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.39@tcp arrived at 1782621963 with bad export cookie 5549677351860500966 [12639.683274] LustreError: 423307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12639.714804] LustreError: 424063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.728138] LustreError: 424063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12640.568315] LustreError: 423306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782621964 with bad export cookie 5549677351860500343 [12640.572445] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12640.573217] LustreError: 423306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12641.762071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12641.768428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12641.776747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12644.832542] LustreError: 423326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12644.833085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12644.837574] LustreError: 423326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12645.856891] LustreError: 424063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.861921] LustreError: 424063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12646.690604] Lustre: server umount lustre-OST0000 complete [12649.361371] Lustre: server umount lustre-MDT0001 complete [12653.714523] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12654.961274] Key type lgssc unregistered [12655.128482] LNet: 427006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12655.132477] LNetError: 427006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12655.142349] LNet: Removed LNI 192.168.201.139@tcp [12655.524190] Key type .llcrypt unregistered [12655.525916] Key type ._llcrypt unregistered [12666.109409] Key type ._llcrypt registered [12666.110358] Key type .llcrypt registered [12666.158454] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12666.552115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12666.601572] alg: No test for adler32 (adler32-zlib) [12667.502208] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12667.612339] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12669.223198] Key type lgssc registered [12669.720089] Lustre: Echo OBD driver; http://www.lustre.org/ [12673.067898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12673.082169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.266952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12675.972609] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12679.450535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12679.589386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12681.050787] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12683.649584] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12685.682701] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12688.165148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12688.294560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12690.436377] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12693.279526] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12699.558258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12705.250920] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12705.254387] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.259971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12708.872925] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [12709.860432] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.061414] Lustre: server umount lustre-MDT0001 complete [12714.977102] LustreError: 428400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12714.977168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12714.977498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12714.981821] LustreError: 428400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12714.987356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12714.997190] Lustre: Skipped 2 previous similar messages [12719.387640] Lustre: server umount lustre-OST0000 complete [12720.096131] LustreError: 428401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.032952] Lustre: server umount lustre-MDT0000 complete [12726.337662] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12727.500754] Key type lgssc unregistered [12727.657029] LNet: 432102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12727.660649] LNetError: 432102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12727.669372] LNet: Removed LNI 192.168.201.139@tcp [12728.042233] Key type .llcrypt unregistered [12728.043715] Key type ._llcrypt unregistered [12737.781578] Key type ._llcrypt registered [12737.783433] Key type .llcrypt registered [12737.826205] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [12744.337888] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12744.784649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12744.796818] alg: No test for adler32 (adler32-zlib) [12745.669107] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12745.769394] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12747.367213] Key type lgssc registered [12747.780759] Lustre: Echo OBD driver; http://www.lustre.org/ [12761.190328] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12767.773282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12767.781123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12768.875875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12768.888991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12768.928729] Lustre: lustre-MDT0000: new disk, initializing [12768.956038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12768.966659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12770.519990] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12776.921339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12777.020483] Lustre: lustre-OST0000: new disk, initializing [12777.022545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12777.024744] Lustre: Skipped 1 previous similar message [12777.027949] Lustre: 437107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12777.055337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12777.441176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12777.445270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12777.459272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12779.278174] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12783.364870] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12789.601712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12789.658120] Lustre: 436087:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12789.707884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12789.759364] Lustre: lustre-MDT0001: new disk, initializing [12789.820421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12789.840101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12789.843949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12791.400880] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12795.350437] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12795.427774] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12796.893285] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12798.013292] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [12802.017609] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12802.018381] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12802.020911] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12802.026848] Lustre: Skipped 1 previous similar message [12803.594438] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.39@tcp (stopping) [12803.597630] Lustre: Skipped 1 previous similar message [12807.137249] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12807.139860] Lustre: Skipped 1 previous similar message [12812.257227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12812.259829] Lustre: Skipped 2 previous similar messages [12814.303266] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12814.359258] Lustre: server umount lustre-MDT0001 complete [12822.497177] LustreError: 436094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12822.503792] LustreError: 436094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12827.616656] LustreError: 439415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.616922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12827.628081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12827.630946] Lustre: Skipped 1 previous similar message [12832.737604] LustreError: 436099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.911459] Lustre: server umount lustre-OST0000 complete [12840.380611] Lustre: server umount lustre-MDT0000 complete [12845.278753] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12846.527092] Key type lgssc unregistered [12846.682461] LNet: 440856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12846.688958] LNetError: 440856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12846.705523] LNet: Removed LNI 192.168.201.139@tcp [12847.054463] Key type .llcrypt unregistered [12847.056201] Key type ._llcrypt unregistered [12861.766586] Key type ._llcrypt registered [12861.768185] Key type .llcrypt registered [12861.823342] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [12861.940425] Key type .llcrypt unregistered [12861.941924] Key type ._llcrypt unregistered [12869.415631] Key type ._llcrypt registered [12869.419173] Key type .llcrypt registered [12869.486361] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [12869.903594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12869.968429] alg: No test for adler32 (adler32-zlib) [12870.854959] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [12870.957673] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [12872.551117] Key type lgssc registered [12873.022778] Lustre: Echo OBD driver; http://www.lustre.org/ [12883.031255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12883.036807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12884.141642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12884.154618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12884.194550] Lustre: lustre-MDT0000: new disk, initializing [12884.223500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12884.233738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12885.686666] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12890.382654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12890.441945] Lustre: 444203:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12890.479865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12890.483047] Lustre: Skipped 1 previous similar message [12890.519929] Lustre: lustre-MDT0001: new disk, initializing [12890.559569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12890.575403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12890.579637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12892.409821] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12896.697448] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12898.704679] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12901.333905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12901.440231] Lustre: lustre-OST0000: new disk, initializing [12901.442378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12901.445590] Lustre: 446170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12901.478043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12902.138934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12902.143230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12902.177594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12903.771618] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12907.904545] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12911.074044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12911.082461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.092542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12912.610190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12912.611905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12912.615933] Lustre: Skipped 1 previous similar message [12912.621620] Lustre: Skipped 1 previous similar message [12916.778975] Lustre: server umount lustre-MDT0000 complete [12918.273097] LustreError: 444196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622242 with bad export cookie 3904210202765896823 [12918.276255] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12918.278221] LustreError: 444196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12918.403829] Lustre: server umount lustre-MDT0001 complete [12929.001401] Lustre: server umount lustre-OST0000 complete [12932.848956] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:50:56 (1782622256) [12936.271331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12936.452660] LustreError: 448195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12936.460595] LustreError: 448195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12936.480594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12937.978828] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12941.319325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12941.487819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12943.095948] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12946.022500] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12948.060414] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12963.554866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12963.631058] Lustre: Found index 0 for lustre-OST0000, updating log [12963.692659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12965.657299] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [12968.138407] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12982.123440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12987.363187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12987.363625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12987.364318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12987.364323] Lustre: Skipped 1 previous similar message [12987.380853] Lustre: Skipped 1 previous similar message [12991.579444] Lustre: server umount lustre-OST0000 complete [13004.519573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13004.520400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13004.526865] Lustre: Skipped 1 previous similar message [13004.528805] Lustre: Skipped 3 previous similar messages [13009.888983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13009.894684] Lustre: Skipped 2 previous similar messages [13010.654331] Lustre: server umount lustre-MDT0000 complete [13012.118168] LustreError: 448176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622336 with bad export cookie 3904210202765898202 [13012.121340] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13012.122771] LustreError: 448176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13012.247172] Lustre: server umount lustre-MDT0001 complete [13015.710250] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:52:19 (1782622339) [13031.711236] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [13595.858038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13596.033067] LustreError: 454630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13596.058192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13597.350655] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13600.016388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13600.132174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13601.418213] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13603.585146] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13605.231344] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13620.450282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13620.535263] Lustre: Found index 0 for lustre-OST0000, updating log [13620.634239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13622.679139] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13623.719019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13625.119790] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13626.785972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.791666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13631.888724] Lustre: server umount lustre-OST0000 complete [13633.101282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13633.105276] Lustre: Skipped 1 previous similar message [13639.253054] Lustre: server umount lustre-MDT0000 complete [13639.265512] LustreError: 454631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.272774] LustreError: 454631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13640.288043] LustreError: 454630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13640.293039] LustreError: 454630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13640.439987] LustreError: 456543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622964 with bad export cookie 3904210202765899273 [13640.442467] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13640.443745] LustreError: 456543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13640.550652] Lustre: server umount lustre-MDT0001 complete [13657.560107] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13659.423214] LNet: 1 local NIs in recovery (showing 1): 192.168.201.139@tcp [13660.589881] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13671.797118] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:03:15 (1782622995) [13677.343574] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13678.294059] Key type lgssc unregistered [13683.551396] LNet: 461282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13683.554141] LNetError: 461282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13752.165458] LNet: Removed LNI 192.168.201.139@tcp [13752.542207] Key type .llcrypt unregistered [13752.543531] Key type ._llcrypt unregistered [13759.556203] Key type ._llcrypt registered [13759.558198] Key type .llcrypt registered [13759.594748] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13760.002962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13760.013144] alg: No test for adler32 (adler32-zlib) [13760.872950] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [13760.963756] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13762.551183] Key type lgssc registered [13762.935388] Lustre: Echo OBD driver; http://www.lustre.org/ [13764.426796] LNet: Added LNI 192.168.201.139@tcp100 [8/256/0/180] [13764.729148] LNet: Added LNI 192.168.201.139@tcp101 [8/256/0/180] [13765.020323] LNet: Added LNI 192.168.201.139@tcp102 [8/256/0/180] [13765.323622] LNet: Added LNI 192.168.201.139@tcp103 [8/256/0/180] [13765.630716] LNet: Added LNI 192.168.201.139@tcp104 [8/256/0/180] [13765.943712] LNet: Added LNI 192.168.201.139@tcp105 [8/256/0/180] [13766.270449] LNet: Added LNI 192.168.201.139@tcp106 [8/256/0/180] [13766.578664] LNet: Added LNI 192.168.201.139@tcp107 [8/256/0/180] [13766.878105] LNet: Added LNI 192.168.201.139@tcp108 [8/256/0/180] [13767.168379] LNet: Added LNI 192.168.201.139@tcp109 [8/256/0/180] [13767.472992] LNet: Added LNI 192.168.201.139@tcp110 [8/256/0/180] [13767.767667] LNet: Added LNI 192.168.201.139@tcp111 [8/256/0/180] [13768.058774] LNet: Added LNI 192.168.201.139@tcp112 [8/256/0/180] [13768.351264] LNet: Added LNI 192.168.201.139@tcp113 [8/256/0/180] [13768.655191] LNet: Added LNI 192.168.201.139@tcp114 [8/256/0/180] [13768.964594] LNet: Added LNI 192.168.201.139@tcp115 [8/256/0/180] [13769.281790] LNet: Added LNI 192.168.201.139@tcp116 [8/256/0/180] [13769.607982] LNet: Added LNI 192.168.201.139@tcp117 [8/256/0/180] [13769.916274] LNet: Added LNI 192.168.201.139@tcp118 [8/256/0/180] [13770.238079] LNet: Added LNI 192.168.201.139@tcp119 [8/256/0/180] [13770.542339] LNet: Added LNI 192.168.201.139@tcp120 [8/256/0/180] [13770.852304] LNet: Added LNI 192.168.201.139@tcp121 [8/256/0/180] [13771.149440] LNet: Added LNI 192.168.201.139@tcp122 [8/256/0/180] [13771.445679] LNet: Added LNI 192.168.201.139@tcp123 [8/256/0/180] [13771.749157] LNet: Added LNI 192.168.201.139@tcp124 [8/256/0/180] [13772.083856] LNet: Added LNI 192.168.201.139@tcp125 [8/256/0/180] [13772.392586] LNet: Added LNI 192.168.201.139@tcp126 [8/256/0/180] [13772.687206] LNet: Added LNI 192.168.201.139@tcp127 [8/256/0/180] [13773.000165] LNet: Added LNI 192.168.201.139@tcp128 [8/256/0/180] [13773.320637] LNet: Added LNI 192.168.201.139@tcp129 [8/256/0/180] [13773.637956] LNet: Added LNI 192.168.201.139@tcp130 [8/256/0/180] [13773.946597] LNet: Added LNI 192.168.201.139@tcp131 [8/256/0/180] [13774.240434] LNet: Added LNI 192.168.201.139@tcp132 [8/256/0/180] [13774.535022] LNet: Added LNI 192.168.201.139@tcp133 [8/256/0/180] [13774.827855] LNet: Added LNI 192.168.201.139@tcp134 [8/256/0/180] [13775.123908] LNet: Added LNI 192.168.201.139@tcp135 [8/256/0/180] [13777.393397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13777.398612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13778.436171] Lustre: 464282:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [13778.462668] LNetError: 464309:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13778.465743] LNetError: Protocol error connecting to 192.168.201.139@tcp120 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13778.468740] LNetError: Skipped 1 previous similar message [13778.528646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13778.540163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13778.582795] Lustre: lustre-MDT0000: new disk, initializing [13778.607194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13778.613262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13779.423345] LNet: 4 local NIs in recovery (showing 4): 192.168.201.139@tcp120, 192.168.201.139@tcp125, 192.168.201.139@tcp126, 192.168.201.139@tcp127 [13779.427602] LNetError: 461744:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13779.430741] LNetError: 461744:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13779.433627] LNetError: Protocol error connecting to 192.168.201.139@tcp120 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13779.438575] LNetError: Skipped 12 previous similar messages [13780.096119] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13781.473719] LNetError: 464304:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13781.478485] LNetError: 464304:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13781.480589] LNetError: Protocol error connecting to 192.168.201.139@tcp127 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13784.154491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13784.207599] Lustre: 464298:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13784.229789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13784.231741] Lustre: Skipped 1 previous similar message [13784.268277] Lustre: lustre-MDT0001: new disk, initializing [13784.279788] Lustre: 465101:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [13784.282286] Lustre: 465101:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13784.318645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13784.333756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13784.336964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13785.568435] LNetError: 464303:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13785.571631] LNetError: Protocol error connecting to 192.168.201.139@tcp127 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13785.864043] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13786.288720] Lustre: 465433:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [13786.291385] Lustre: 465433:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13789.496505] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13791.217361] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13793.435679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13793.555291] Lustre: lustre-OST0000: new disk, initializing [13793.557779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13793.560380] Lustre: 466272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13793.591186] Lustre: 466277:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [13793.593748] Lustre: 466277:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13793.611930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13793.760244] LNetError: 464302:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13793.763465] LNetError: Protocol error connecting to 192.168.201.139@tcp127 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13795.495946] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13799.099186] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13801.362321] Lustre: server umount lustre-OST0000 complete [13804.000672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13804.001445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13804.004822] Lustre: Skipped 1 previous similar message [13804.015130] Lustre: 467416:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.139@tcp: can't add NID 192.168.201.139@tcp130: rc = -75 [13804.017444] Lustre: 467416:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13808.096938] LNetError: 464311:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.139 [13808.100390] LNetError: Protocol error connecting to 192.168.201.139@tcp123 at host 192.168.201.139:988: is it running a compatible version of Lustre? [13808.724760] Lustre: server umount lustre-MDT0000 complete [13809.938098] LustreError: 464292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623134 with bad export cookie 1550988067014180207 [13809.940773] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13809.941042] LustreError: 464292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13810.046610] Lustre: server umount lustre-MDT0001 complete [13813.475501] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13814.389185] Key type lgssc unregistered [13814.535120] LNet: 468175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13814.537705] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13814.539636] LNet: Removed LNI 192.168.201.139@tcp [13815.583151] LNet: Removed LNI 192.168.201.139@tcp100 [13815.584696] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13815.587107] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13815.589342] LNet: Removed LNI 192.168.201.139@tcp101 [13816.607123] LNet: Removed LNI 192.168.201.139@tcp102 [13816.608838] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13816.611591] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13816.614591] LNet: Removed LNI 192.168.201.139@tcp103 [13817.631183] LNet: Removed LNI 192.168.201.139@tcp104 [13818.655159] LNet: Removed LNI 192.168.201.139@tcp105 [13818.656942] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13818.659867] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13818.663085] LNet: Removed LNI 192.168.201.139@tcp106 [13819.679162] LNet: Removed LNI 192.168.201.139@tcp107 [13820.703168] LNet: Removed LNI 192.168.201.139@tcp108 [13821.727141] LNet: Removed LNI 192.168.201.139@tcp109 [13822.751171] LNet: Removed LNI 192.168.201.139@tcp110 [13822.753019] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13822.756120] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13822.760089] LNet: Removed LNI 192.168.201.139@tcp111 [13823.775170] LNet: Removed LNI 192.168.201.139@tcp112 [13824.799169] LNet: Removed LNI 192.168.201.139@tcp113 [13825.823136] LNet: Removed LNI 192.168.201.139@tcp114 [13826.847161] LNet: Removed LNI 192.168.201.139@tcp115 [13827.871128] LNet: Removed LNI 192.168.201.139@tcp116 [13828.895115] LNet: Removed LNI 192.168.201.139@tcp117 [13829.919111] LNet: Removed LNI 192.168.201.139@tcp118 [13830.943154] LNet: Removed LNI 192.168.201.139@tcp119 [13830.944368] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13830.946659] LNetError: 468175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13830.948777] LNet: Removed LNI 192.168.201.139@tcp120 [13831.967158] LNet: Removed LNI 192.168.201.139@tcp121 [13832.991154] LNet: Removed LNI 192.168.201.139@tcp122 [13834.015077] LNet: Removed LNI 192.168.201.139@tcp123 [13835.039157] LNet: Removed LNI 192.168.201.139@tcp124 [13836.063140] LNet: Removed LNI 192.168.201.139@tcp125 [13837.087159] LNet: Removed LNI 192.168.201.139@tcp126 [13837.088725] LNet: Removed LNI 192.168.201.139@tcp127 [13838.111134] LNet: Removed LNI 192.168.201.139@tcp128 [13839.135132] LNet: Removed LNI 192.168.201.139@tcp129 [13839.136516] LNet: Removed LNI 192.168.201.139@tcp130 [13839.138043] LNet: Removed LNI 192.168.201.139@tcp131 [13839.139535] LNet: Removed LNI 192.168.201.139@tcp132 [13839.140788] LNet: Removed LNI 192.168.201.139@tcp133 [13839.141933] LNet: Removed LNI 192.168.201.139@tcp134 [13839.149470] LNet: Removed LNI 192.168.201.139@tcp135 [13839.437664] Key type .llcrypt unregistered [13839.438649] Key type ._llcrypt unregistered [13846.867671] Key type ._llcrypt registered [13846.868685] Key type .llcrypt registered [13846.907649] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_hostid [13852.613666] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13852.909651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13852.915419] alg: No test for adler32 (adler32-zlib) [13853.758032] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [13853.836812] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13855.415170] Key type lgssc registered [13855.741155] Lustre: Echo OBD driver; http://www.lustre.org/ [13866.138514] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:06:30 (1782623190) [13871.909430] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing unload_modules_local [13872.855177] Key type lgssc unregistered [13872.978331] LNet: 472233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13872.981113] LNetError: 472233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13872.990406] LNet: Removed LNI 192.168.201.139@tcp [13873.275336] Key type .llcrypt unregistered [13873.276295] Key type ._llcrypt unregistered [13879.552402] Key type ._llcrypt registered [13879.553356] Key type .llcrypt registered [13879.591993] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [13880.020727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13880.065181] alg: No test for adler32 (adler32-zlib) [13880.921690] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [13881.018978] LNet: Added LNI 192.168.201.139@tcp [8/256/0/180] [13882.599254] Key type lgssc registered [13882.998868] Lustre: Echo OBD driver; http://www.lustre.org/ [13885.814469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13885.820704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13886.912254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13886.922110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13886.961312] Lustre: lustre-MDT0000: new disk, initializing [13886.982836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13886.991148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13888.320635] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13892.221922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13892.263089] Lustre: 473419:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13892.285388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13892.287092] Lustre: Skipped 1 previous similar message [13892.322110] Lustre: lustre-MDT0001: new disk, initializing [13892.355180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13892.366777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13892.370155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13893.716329] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13897.393260] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13899.139267] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13901.427830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13901.535505] Lustre: lustre-OST0000: new disk, initializing [13901.538229] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13901.542097] Lustre: 475387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13901.574889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13903.035837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13903.040394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13903.079783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13903.604884] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [13907.346971] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13908.487231] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing lnet_if_list [13909.980935] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13909.994501] LNet: Added LNI 192.168.201.139@tcp10 [8/256/0/180] [13911.672274] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13911.683087] LNet: Added LNI 192.168.201.139@tcp11 [8/256/0/180] [13912.952098] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13912.962765] LNet: Added LNI 192.168.201.139@tcp12 [8/256/0/180] [13914.186150] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13914.196328] LNet: Added LNI 192.168.201.139@tcp13 [8/256/0/180] [13915.610607] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13915.621610] LNet: Added LNI 192.168.201.139@tcp14 [8/256/0/180] [13916.946406] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13916.956421] LNet: Added LNI 192.168.201.139@tcp15 [8/256/0/180] [13918.296572] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13918.306506] LNet: Added LNI 192.168.201.139@tcp16 [8/256/0/180] [13919.646800] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13919.656552] LNet: Added LNI 192.168.201.139@tcp17 [8/256/0/180] [13921.105807] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13921.120036] LNet: Added LNI 192.168.201.139@tcp18 [8/256/0/180] [13922.579637] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13922.591530] LNet: Added LNI 192.168.201.139@tcp19 [8/256/0/180] [13924.103683] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13924.114740] LNet: Added LNI 192.168.201.139@tcp20 [8/256/0/180] [13925.598129] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13925.611731] LNet: Added LNI 192.168.201.139@tcp21 [8/256/0/180] [13927.183576] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13927.195563] LNet: Added LNI 192.168.201.139@tcp22 [8/256/0/180] [13928.746712] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13928.755262] LNet: Added LNI 192.168.201.139@tcp23 [8/256/0/180] [13930.314436] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13930.324870] LNet: Added LNI 192.168.201.139@tcp24 [8/256/0/180] [13931.908436] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13931.918384] LNet: Added LNI 192.168.201.139@tcp25 [8/256/0/180] [13933.535864] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13933.545165] LNet: Added LNI 192.168.201.139@tcp26 [8/256/0/180] [13935.198950] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13935.210028] LNet: Added LNI 192.168.201.139@tcp27 [8/256/0/180] [13936.929516] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13936.940421] LNet: Added LNI 192.168.201.139@tcp28 [8/256/0/180] [13938.687230] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13938.698575] LNet: Added LNI 192.168.201.139@tcp29 [8/256/0/180] [13940.481816] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13940.493056] LNet: Added LNI 192.168.201.139@tcp30 [8/256/0/180] [13942.185468] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13942.195432] LNet: Added LNI 192.168.201.139@tcp31 [8/256/0/180] [13943.985656] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13943.994814] LNet: Added LNI 192.168.201.139@tcp32 [8/256/0/180] [13945.811093] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13945.820471] LNet: Added LNI 192.168.201.139@tcp33 [8/256/0/180] [13947.762220] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13947.772492] LNet: Added LNI 192.168.201.139@tcp34 [8/256/0/180] [13949.659394] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13949.670540] LNet: Added LNI 192.168.201.139@tcp35 [8/256/0/180] [13951.706167] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13951.716098] LNet: Added LNI 192.168.201.139@tcp36 [8/256/0/180] [13953.661628] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13953.674530] LNet: Added LNI 192.168.201.139@tcp37 [8/256/0/180] [13955.632512] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13955.641360] LNet: Added LNI 192.168.201.139@tcp38 [8/256/0/180] [13957.753470] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13957.764512] LNet: Added LNI 192.168.201.139@tcp39 [8/256/0/180] [13959.858806] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13959.870060] LNet: Added LNI 192.168.201.139@tcp40 [8/256/0/180] [13961.946207] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13961.958118] LNet: Added LNI 192.168.201.139@tcp41 [8/256/0/180] [13964.034939] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13964.045300] LNet: Added LNI 192.168.201.139@tcp42 [8/256/0/180] [13966.167154] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13966.177357] LNet: Added LNI 192.168.201.139@tcp43 [8/256/0/180] [13968.349088] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13968.359512] LNet: Added LNI 192.168.201.139@tcp44 [8/256/0/180] [13970.488909] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13970.499856] LNet: Added LNI 192.168.201.139@tcp45 [8/256/0/180] [13972.751992] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13972.762911] LNet: Added LNI 192.168.201.139@tcp46 [8/256/0/180] [13975.014429] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13975.024151] LNet: Added LNI 192.168.201.139@tcp47 [8/256/0/180] [13977.792317] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13977.804190] LNet: Added LNI 192.168.201.139@tcp48 [8/256/0/180] [13980.532931] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13980.544134] LNet: Added LNI 192.168.201.139@tcp49 [8/256/0/180] [13983.339407] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13983.351371] LNet: Added LNI 192.168.201.139@tcp50 [8/256/0/180] [13994.975746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13994.978574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13994.982078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13995.488238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13995.489058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13995.490393] Lustre: Skipped 1 previous similar message [13995.493736] Lustre: Skipped 1 previous similar message [13996.633893] Lustre: server umount lustre-MDT0000 complete [13997.878031] LustreError: 473414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623322 with bad export cookie 12892639634951765799 [13997.880575] LustreError: MGC192.168.201.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13997.881442] LustreError: 473414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13997.990418] Lustre: server umount lustre-MDT0001 complete [14004.927149] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing load_modules_local [14007.688179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14008.735195] LustreError: 499557:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [14008.735461] LustreError: 472694:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000010bce883 x1869215909885568/t0(0) o250->MGC192.168.201.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14008.837617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14010.118254] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14012.841377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14012.925167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14012.956943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14014.260645] Lustre: DEBUG MARKER: oleg139-server.virtnet: executing set_default_debug -1 all [14015.013569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14015.967109] Lustre: 472698:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782623324/real 1782623324] req@0000000005087bdb x1869215909885184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782623340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14016.554170] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14018.056137] Lustre: 472695:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782623326/real 1782623326] req@00000000c9a0c0ee x1869215909885312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782623342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14018.384552] Lustre: DEBUG MARKER: oleg139-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14021.087196] Lustre: 472696:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782623329/real 1782623329] req@00000000958bc6cb x1869215909885440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782623345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14024.160087] Lustre: 472697:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782623332/real 1782623332] req@000000009a6e789b x1869215909885696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782623348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14028.256330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14028.259687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14028.266814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)