[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 568247491 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003064] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010228] LSM: Security Framework initializing [ 0.011109] Yama: becoming mindful. [ 0.012061] SELinux: Initializing. [ 0.013086] *** VALIDATE selinux *** [ 0.021205] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025741] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027089] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028103] *** VALIDATE tmpfs *** [ 0.029399] *** VALIDATE proc *** [ 0.030239] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.033027] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034117] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040363] debug: unmapping init [mem 0xffffffff8cc59000-0xffffffff8cc60fff] [ 0.042177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051313] rcu: Hierarchical SRCU implementation. [ 0.053442] smp: Bringing up secondary CPUs ... [ 0.054484] x86: Booting SMP configuration: [ 0.055030] .... node #0, CPUs: #1 #2 #3 [ 0.058369] smp: Brought up 1 node, 4 CPUs [ 0.060011] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.116009] node 0 deferred pages initialised in 53ms [ 0.118103] devtmpfs: initialized [ 0.119295] x86/mm: Memory block size: 128MB [ 0.122810] gcov: version magic: 0x41383552 [ 0.125301] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.127078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128306] pinctrl core: initialized pinctrl subsystem [ 0.129348] [ 0.129936] ************************************************************* [ 0.132017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134135] ** ** [ 0.136014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139013] ** ** [ 0.142013] ** This means that this kernel is built to expose internal ** [ 0.145014] ** IOMMU data structures, which may compromise security on ** [ 0.147010] ** your system. ** [ 0.150068] ** ** [ 0.152008] ** If you see this message and you are not debugging the ** [ 0.154008] ** kernel, report this immediately to your vendor! ** [ 0.155007] ** ** [ 0.157008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158006] ************************************************************* [ 0.160716] NET: Registered protocol family 16 [ 0.162407] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166130] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170031] cpuidle: using governor menu [ 0.171645] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173342] PCI: Using configuration type 1 for base access [ 0.175106] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.182032] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184198] cryptd: max_cpu_qlen set to 1000 [ 0.187204] ACPI: Added _OSI(Module Device) [ 0.188010] ACPI: Added _OSI(Processor Device) [ 0.190011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194153] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198497] ACPI: Interpreter enabled [ 0.200043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.201008] ACPI: Using IOAPIC for interrupt routing [ 0.202088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217996] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.223024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.227160] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233581] acpiphp: Slot [2] registered [ 0.235116] acpiphp: Slot [5] registered [ 0.236174] acpiphp: Slot [6] registered [ 0.238120] acpiphp: Slot [7] registered [ 0.240164] acpiphp: Slot [8] registered [ 0.241142] acpiphp: Slot [9] registered [ 0.243117] acpiphp: Slot [10] registered [ 0.245161] acpiphp: Slot [3] registered [ 0.246095] acpiphp: Slot [4] registered [ 0.247188] acpiphp: Slot [11] registered [ 0.249121] acpiphp: Slot [12] registered [ 0.251069] acpiphp: Slot [13] registered [ 0.252084] acpiphp: Slot [14] registered [ 0.253109] acpiphp: Slot [15] registered [ 0.255087] acpiphp: Slot [16] registered [ 0.256086] acpiphp: Slot [17] registered [ 0.258102] acpiphp: Slot [18] registered [ 0.259103] acpiphp: Slot [19] registered [ 0.261109] acpiphp: Slot [20] registered [ 0.263162] acpiphp: Slot [21] registered [ 0.264117] acpiphp: Slot [22] registered [ 0.266100] acpiphp: Slot [23] registered [ 0.268103] acpiphp: Slot [24] registered [ 0.270108] acpiphp: Slot [25] registered [ 0.271105] acpiphp: Slot [26] registered [ 0.273097] acpiphp: Slot [27] registered [ 0.275115] acpiphp: Slot [28] registered [ 0.276146] acpiphp: Slot [29] registered [ 0.278092] acpiphp: Slot [30] registered [ 0.279158] acpiphp: Slot [31] registered [ 0.281065] PCI host bridge to bus 0000:00 [ 0.282013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.288040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.293026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.295172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298008] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301221] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309903] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313540] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322331] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.324531] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328665] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342844] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354590] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394013] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.405017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.410018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.438659] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.446014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.473021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.483130] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.490018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.500016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.528019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.538943] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.545017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.556022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.592000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.596016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.600017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.619014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.627105] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.632715] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.634425] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.636379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637114] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.641226] iommu: Default domain type: Passthrough [ 0.642392] SCSI subsystem initialized [ 0.644113] ACPI: bus type USB registered [ 0.645097] usbcore: registered new interface driver usbfs [ 0.647088] usbcore: registered new interface driver hub [ 0.649083] usbcore: registered new device driver usb [ 0.651190] pps_core: LinuxPPS API ver. 1 registered [ 0.653011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.655052] PTP clock support registered [ 0.657067] EDAC MC: Ver: 3.0.0 [ 0.658125] PCI: Using ACPI for IRQ routing [ 0.659589] NetLabel: Initializing [ 0.660007] NetLabel: domain hash size = 128 [ 0.660862] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.662067] NetLabel: unlabeled traffic allowed by default [ 0.663103] vgaarb: loaded [ 0.664328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.666015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.671238] clocksource: Switched to clocksource kvm-clock [ 0.771626] VFS: Disk quotas dquot_6.6.0 [ 0.773089] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.775588] *** VALIDATE ramfs *** [ 0.776634] *** VALIDATE hugetlbfs *** [ 0.777962] pnp: PnP ACPI init [ 0.780095] pnp: PnP ACPI: found 6 devices [ 0.794045] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796639] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.798801] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.800290] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.802568] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.804406] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.807284] NET: Registered protocol family 2 [ 0.809235] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.813651] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816333] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820541] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.822817] TCP: Hash tables configured (established 65536 bind 65536) [ 0.824677] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.826428] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.828144] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.829892] NET: Registered protocol family 1 [ 0.831622] RPC: Registered named UNIX socket transport module. [ 0.834178] RPC: Registered udp transport module. [ 0.835432] RPC: Registered tcp transport module. [ 0.836857] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.838289] NET: Registered protocol family 44 [ 0.839687] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.841081] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.843061] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.844666] PCI: CLS 0 bytes, default 64 [ 0.846681] Unpacking initramfs... [ 4.140209] debug: unmapping init [mem 0xffff9754fcc54000-0xffff9754fffbffff] [ 4.155436] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.162688] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.173480] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.465448] Initialise system trusted keyrings [ 6.470811] Key type blacklist registered [ 6.473875] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.487412] zbud: loaded [ 6.491949] *** VALIDATE nfs *** [ 6.493402] *** VALIDATE nfs4 *** [ 6.502976] pstore: using deflate compression [ 6.507748] Platform Keyring initialized [ 6.734513] NET: Registered protocol family 38 [ 6.738975] Key type asymmetric registered [ 6.740967] Asymmetric key parser 'x509' registered [ 6.744169] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.751610] io scheduler mq-deadline registered [ 6.758251] io scheduler kyber registered [ 6.766680] io scheduler bfq registered [ 6.769865] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.775279] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.782842] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.788596] ACPI: Power Button [PWRF] [ 6.807844] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.825142] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.846873] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.861991] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.908831] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.965166] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.005060] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.017261] Non-volatile memory driver v1.3 [ 7.021571] Linux agpgart interface v0.103 [ 7.084806] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 7.096501] vda: detected capacity change from 0 to 74698752 [ 7.141239] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.146205] vdb: detected capacity change from 0 to 1073741824 [ 7.180706] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.187242] vdc: detected capacity change from 0 to 2621440000 [ 7.228860] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.235155] vdd: detected capacity change from 0 to 2621440000 [ 7.291552] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.297268] vde: detected capacity change from 0 to 4294967296 [ 7.359584] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.368061] vdf: detected capacity change from 0 to 4294967296 [ 7.385860] libphy: Fixed MDIO Bus: probed [ 7.409218] usbcore: registered new interface driver usbserial_generic [ 7.411427] usbserial: USB Serial support registered for generic [ 7.414729] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.424539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.431160] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.440842] mousedev: PS/2 mouse device common for all mice [ 7.446842] rtc_cmos 00:05: RTC can wake from S4 [ 7.455102] rtc_cmos 00:05: registered as rtc0 [ 7.459744] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.468333] intel_pstate: CPU model not supported [ 7.481347] hid: raw HID events driver (C) Jiri Kosina [ 7.485834] usbcore: registered new interface driver usbhid [ 7.487800] usbhid: USB HID core driver [ 7.493361] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.496230] drop_monitor: Initializing network drop monitor service [ 7.502183] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.503215] Initializing XFRM netlink socket [ 7.512740] NET: Registered protocol family 10 [ 7.515102] Segment Routing with IPv6 [ 7.516035] NET: Registered protocol family 17 [ 7.520724] mpls_gso: MPLS GSO support [ 7.535480] RAS: Correctable Errors collector initialized. [ 7.537616] AVX version of gcm_enc/dec engaged. [ 7.544214] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.552218] AES CTR mode by8 optimization enabled [ 7.699362] sched_clock: Marking stable (7699280607, 0)->(8605455693, -906175086) [ 7.703211] registered taskstats version 1 [ 7.706386] Loading compiled-in X.509 certificates [ 7.709371] zswap: loaded using pool lzo/zbud [ 7.777495] Key type big_key registered [ 7.799695] Key type encrypted registered [ 7.806208] ima: No TPM chip found, activating TPM-bypass! [ 7.811366] ima: Allocated hash algorithm: sha1 [ 7.816638] ima: No architecture policies found [ 7.827411] evm: Initialising EVM extended attributes: [ 7.836484] evm: security.selinux [ 7.843389] evm: security.ima [ 7.845639] evm: security.capability [ 7.848563] evm: HMAC attrs: 0x1 [ 7.853403] rtc_cmos 00:05: setting system clock to 2026-08-08 10:33:43 UTC (1786185223) [ 7.866968] debug: unmapping init [mem 0xffffffff8dc03000-0xffffffff8ddfffff] [ 7.872317] debug: unmapping init [mem 0xffffffff8c982000-0xffffffff8cc58fff] [ 7.890082] Write protecting the kernel read-only data: 28672k [ 7.911188] debug: unmapping init [mem 0xffffffff8b003000-0xffffffff8b1fffff] [ 7.934145] debug: unmapping init [mem 0xffffffff8b914000-0xffffffff8b9fffff] [ 8.008461] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.038585] systemd[1]: Detected virtualization kvm. [ 8.043776] systemd[1]: Detected architecture x86-64. [ 8.050484] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.108576] systemd[1]: No hostname configured. [ 8.111660] systemd[1]: Set hostname to . [ 8.115243] random: systemd: uninitialized urandom read (16 bytes read) [ 8.126319] systemd[1]: Initializing machine ID from random generator. [ 8.305763] random: ln: uninitialized urandom read (6 bytes read) [ 8.628602] random: systemd: uninitialized urandom read (16 bytes read) [ 8.638974] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 8.667926] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.684977] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.681923] device-mapper: uevent: version 1.0.3 [ 10.690810] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.886932] virtio_net virtio0 ens2: renamed from eth0 [ 13.129922] random: fast init done [ 13.383760] scsi host0: ata_piix [ 13.428284] scsi host1: ata_piix [ 13.433072] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.441383] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.369191] random: crng init done [ 18.375553] random: 7 urandom warning(s) missed due to ratelimiting [ 21.063622] dracut-initqueue[602]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.024956] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.688062] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.988985] SELinux: Disabled at runtime. [ 28.154374] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.179513] systemd[1]: Detected virtualization kvm. [ 28.182863] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.588238] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.596779] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.618535] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.622547] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.628919] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.659814] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.672670] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 30.701663] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.734610] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.940258] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.146399] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.895345] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.026386] EDAC sbridge: Ver: 1.1.2 [ 37.621429] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only[ 38.481377] NFS: Registering the id_resolver key type root support (9s / no limit)[ 38.488204] Key type id_resolver registered [ 38.506387] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 80.488221] hrtimer: interrupt took 4571323 ns [ 95.104664] libcfs: loading out-of-tree module taints kernel. [ 95.154618] Key type ._llcrypt registered [ 95.157461] Key type .llcrypt registered [ 95.277772] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 114.836431] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 116.093825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.114048] alg: No test for adler32 (adler32-zlib) [ 117.626523] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 118.428731] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 120.272216] Key type lgssc registered [ 122.215942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 139.327412] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.022679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 197.983145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 198.061618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.584355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.638379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.838804] Lustre: lustre-MDT0000: new disk, initializing [ 199.967601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.992795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.323712] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.734367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.832150] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.890503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.901105] Lustre: Skipped 1 previous similar message [ 223.099535] Lustre: lustre-MDT0001: new disk, initializing [ 223.219797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.272361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.290092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.910874] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.043232] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.869819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 246.094640] Lustre: lustre-OST0000: new disk, initializing [ 246.097934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 246.102442] Lustre: 8464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 246.154781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.976402] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.482260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.493373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.607992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.396156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.599292] Lustre: lustre-OST0001: new disk, initializing [ 269.608582] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.616983] Lustre: 9537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.758826] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.870462] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.625885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.636301] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.728690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.361561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.216172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.981932] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 308.253498] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 311.836790] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 314.191646] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 316.526526] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 318.079716] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 06:38:52 EDT 2026 [ 335.520822] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 337.225959] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.714252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.715627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.729150] Lustre: Skipped 1 previous similar message [ 351.735460] Lustre: Skipped 2 previous similar messages [ 356.322189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 356.328846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.347514] Lustre: Skipped 1 previous similar message [ 356.351707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.048971] Lustre: server umount lustre-MDT0000 complete [ 362.046986] LustreError: 10412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185577 with bad export cookie 6810881364709041295 [ 362.051857] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.056389] LustreError: 10412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 367.074354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.076107] LustreError: 6533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.078971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.079243] Lustre: Skipped 3 previous similar messages [ 367.092233] Lustre: Skipped 1 previous similar message [ 367.122068] LustreError: 6533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 368.393705] Lustre: server umount lustre-MDT0001 complete [ 372.075330] Lustre: server umount lustre-OST0000 complete [ 375.982821] Lustre: server umount lustre-OST0001 complete [ 383.115494] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 397.801318] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 437.914301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.177910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.204020] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.254243] Lustre: lustre-MDT0000: new disk, initializing [ 438.314604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.326915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 443.145878] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 453.768066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.910947] Lustre: 15102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 453.983891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 453.988160] Lustre: Skipped 1 previous similar message [ 454.099343] Lustre: lustre-MDT0001: new disk, initializing [ 454.222489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.275334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.301134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 458.741418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 468.336646] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 474.106752] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.899683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.284536] Lustre: lustre-OST0000: new disk, initializing [ 482.292780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.302982] Lustre: 17071:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.430811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.552823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 483.558392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 483.699049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.449790] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 499.515382] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 504.769587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 505.001849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 509.959689] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 510.216926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 514.533465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.533973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.539750] LustreError: Skipped 1 previous similar message [ 514.543187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.550530] Lustre: Skipped 1 previous similar message [ 514.553802] Lustre: Skipped 1 previous similar message [ 518.435043] Lustre: server umount lustre-OST0000 complete [ 524.775158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.794733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.799984] Lustre: Skipped 2 previous similar messages [ 528.486613] Lustre: server umount lustre-MDT0000 complete [ 529.892264] LustreError: 15115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.915851] LustreError: 15115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 532.255803] LustreError: 15097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185747 with bad export cookie 6810881364709042835 [ 532.267424] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.270745] LustreError: 15097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 532.551146] Lustre: server umount lustre-MDT0001 complete [ 536.280674] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:42:30 (1786185750) [ 538.707847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 540.865759] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:42:34 (1786185754) [ 548.748701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.174769] LustreError: 19481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.243198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.917214] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 554.469895] LustreError: 19482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.569622] LustreError: 19481:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.038795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.171496] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 577.113798] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 584.027165] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.917146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.535552] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 610.555598] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 618.979262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 618.993384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.999739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.001819] Lustre: Skipped 2 previous similar messages [ 624.726902] Lustre: server umount lustre-OST0000 complete [ 634.828465] Lustre: server umount lustre-MDT0000 complete [ 639.372271] LustreError: 20217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786185855 with bad export cookie 6810881364709043717 [ 639.377040] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.380309] LustreError: 20217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 639.727959] Lustre: server umount lustre-MDT0001 complete [ 652.559230] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 655.887273] Key type lgssc unregistered [ 656.200762] LNet: 23020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.212607] LNetError: 23020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.228069] LNet: Removed LNI 192.168.201.109@tcp [ 657.554238] Key type .llcrypt unregistered [ 657.572060] Key type ._llcrypt unregistered [ 687.212899] Key type ._llcrypt registered [ 687.217202] Key type .llcrypt registered [ 687.372581] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 688.974128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.008809] alg: No test for adler32 (adler32-zlib) [ 690.044875] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 690.362731] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 692.088204] Key type lgssc registered [ 693.523362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 703.203634] loop: module loaded [ 703.330303] loop0: detected capacity change from 0 to 204800000 [ 703.546609] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.554292] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.844912] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.943418] loop0: detected capacity change from 0 to 204800000 [ 710.162172] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.170956] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.180416] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 714.965425] loop0: detected capacity change from 0 to 204800000 [ 720.800866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.836881] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.173563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 722.232700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 722.348563] Lustre: lustre-MDT0000: new disk, initializing [ 722.538252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.563482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 727.407730] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 735.653124] loop1: detected capacity change from 0 to 204800000 [ 741.101647] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.432469] Lustre: lustre-OST07c6: new disk, initializing [ 741.435695] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 741.440308] Lustre: Skipped 1 previous similar message [ 741.448724] Lustre: 26207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 741.532312] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 748.712891] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 750.737996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 750.754083] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 750.810948] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 754.402669] Lustre: 26670:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 754.415448] Lustre: 26670:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 754.421613] Lustre: 26670:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 754.427098] Lustre: 26670:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 754.432413] Lustre: 26670:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 754.437473] Lustre: 26670:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 768.188545] Lustre: 25198:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 768.195329] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.199396] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.204189] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 768.209847] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 768.215130] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.868906] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 772.894546] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.905580] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 772.915605] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.925055] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.934241] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.942000] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.951274] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.958898] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.968473] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.981802] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.994208] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.235268] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 776.243048] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 776.248782] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 776.255734] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.263408] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.272437] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.282642] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 776.291726] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.300733] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.309589] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.318972] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.324659] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.619487] Lustre: 26670:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 781.629653] Lustre: 26670:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 781.635943] Lustre: 26670:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.641698] Lustre: 26670:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.646707] Lustre: 26670:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.652858] Lustre: 26670:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.659616] Lustre: 26670:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 781.666054] Lustre: 26670:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.674200] Lustre: 26670:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.681253] Lustre: 26670:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.687592] Lustre: 26670:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.694188] Lustre: 26670:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 792.636235] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 792.645391] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 792.651503] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.657470] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.663312] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.669759] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.676717] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 792.681767] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.687436] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 792.699805] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.709531] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.716112] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 842.729287] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 842.748208] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.766524] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.547886] Lustre: server umount lustre-OST07c6 complete [ 855.977292] Lustre: server umount lustre-MDT0000 complete [ 875.482672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 879.326958] Key type lgssc unregistered [ 879.691687] LNet: 28767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.701456] LNetError: 28767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.716860] LNet: Removed LNI 192.168.201.109@tcp [ 880.776280] Key type .llcrypt unregistered [ 880.777599] Key type ._llcrypt unregistered [ 901.058451] Key type ._llcrypt registered [ 901.059960] Key type .llcrypt registered [ 901.124131] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 902.407897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.498408] alg: No test for adler32 (adler32-zlib) [ 903.621801] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 903.814892] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 905.512255] Key type lgssc registered [ 906.888702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.409086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.456066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.911617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.975298] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 927.227493] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.261501] LustreError: 30041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 929.430083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.203044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.227452] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 944.499262] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 951.740681] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.679143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 959.255574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.355824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 965.824215] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 973.472466] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.129331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.142857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.169325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.193687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 990.194972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 990.212320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.365469] Lustre: server umount lustre-OST0000 complete [ 1000.419704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.432205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.451037] Lustre: Skipped 1 previous similar message [ 1000.460979] Lustre: Skipped 2 previous similar messages [ 1004.268307] Lustre: server umount lustre-MDT0000 complete [ 1005.552308] LustreError: 30037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.591260] LustreError: 30037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1008.922611] LustreError: 30022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186224 with bad export cookie 13751320944775278926 [ 1008.931133] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1008.931560] LustreError: 30022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1009.211282] Lustre: server umount lustre-MDT0001 complete [ 1023.568961] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1027.145801] Key type lgssc unregistered [ 1027.523759] LNet: 33781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.528557] LNetError: 33781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.555745] LNet: Removed LNI 192.168.201.109@tcp [ 1028.384228] Key type .llcrypt unregistered [ 1028.388171] Key type ._llcrypt unregistered [ 1046.846438] Key type ._llcrypt registered [ 1046.848790] Key type .llcrypt registered [ 1046.988071] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1048.258156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.286873] alg: No test for adler32 (adler32-zlib) [ 1049.430139] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 1049.712846] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1051.368239] Key type lgssc registered [ 1052.548995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.223358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.237311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.767210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1066.640128] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1072.100692] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.138659] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1073.124595] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.807235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.249731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1082.595147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1092.189874] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1098.366185] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.131774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.457901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.535944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1111.681837] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1119.414865] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.492032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.510358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.535443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.564253] Lustre: Skipped 1 previous similar message [ 1134.550398] Lustre: server umount lustre-OST0000 complete [ 1141.734348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.735955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.746594] Lustre: Skipped 2 previous similar messages [ 1141.770707] Lustre: Skipped 1 previous similar message [ 1145.329814] Lustre: server umount lustre-MDT0000 complete [ 1146.851577] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.872799] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1149.234850] LustreError: 35035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186364 with bad export cookie 2343553670567798701 [ 1149.243219] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1149.712960] Lustre: server umount lustre-MDT0001 complete [ 1161.176800] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1163.156585] Key type lgssc unregistered [ 1163.547918] LNet: 38592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.551448] LNetError: 38592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.580259] LNet: Removed LNI 192.168.201.109@tcp [ 1164.854931] Key type .llcrypt unregistered [ 1164.857700] Key type ._llcrypt unregistered [ 1192.259387] Key type ._llcrypt registered [ 1192.266292] Key type .llcrypt registered [ 1192.419692] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1193.258062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.276459] alg: No test for adler32 (adler32-zlib) [ 1194.708119] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 1194.939297] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1196.672209] Key type lgssc registered [ 1198.340546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.513682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.587400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.373638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.800825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1219.571696] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.594992] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1224.677798] LustreError: 40010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.176486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.451633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.570779] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1238.730666] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.486429] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.162363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.596474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.724883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1256.989126] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1265.302341] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1273.296414] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1275.362130] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.377398] Lustre: Skipped 1 previous similar message [ 1275.382604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.399260] Lustre: Skipped 2 previous similar messages [ 1278.387925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1278.759589] Lustre: server umount lustre-MDT0000 complete [ 1280.483961] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1280.507425] LustreError: 40009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1282.345843] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186497 with bad export cookie 3750300343145480612 [ 1282.350765] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.351767] LustreError: 39990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1283.488691] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.515747] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 1285.612340] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.617774] Lustre: Skipped 1 previous similar message [ 1287.649374] LustreError: 41888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.651975] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1287.669238] LustreError: 41888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1287.688666] Lustre: Skipped 1 previous similar message [ 1288.817867] Lustre: server umount lustre-MDT0001 complete [ 1311.039746] Lustre: server umount lustre-OST0000 complete [ 1331.890525] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1335.182450] Key type lgssc unregistered [ 1335.532876] LNet: 43874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.547317] LNetError: 43874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.560256] LNet: Removed LNI 192.168.201.109@tcp [ 1336.528337] Key type .llcrypt unregistered [ 1336.530197] Key type ._llcrypt unregistered [ 1363.696849] Key type ._llcrypt registered [ 1363.699346] Key type .llcrypt registered [ 1363.840194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1364.994545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1365.121194] alg: No test for adler32 (adler32-zlib) [ 1366.246493] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 1366.516207] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1368.208361] Key type lgssc registered [ 1369.697397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1379.443568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1379.454446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.190456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.134525] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1390.579356] LustreError: 45293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.606859] LustreError: 45293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1395.683066] LustreError: 45292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.921563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.425861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1403.158134] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1412.120207] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.777629] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1426.791718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1427.307147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.636633] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1442.617315] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1450.881478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1461.229820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.251859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.258766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.268203] Lustre: Skipped 1 previous similar message [ 1461.300109] Lustre: Skipped 1 previous similar message [ 1463.769042] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1465.811744] Lustre: server umount lustre-OST0000 complete [ 1485.793761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.807277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.824268] Lustre: Skipped 1 previous similar message [ 1485.853954] Lustre: Skipped 1 previous similar message [ 1490.913418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.926788] Lustre: Skipped 1 previous similar message [ 1496.042426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.047143] Lustre: Skipped 1 previous similar message [ 1499.104882] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1499.437405] Lustre: server umount lustre-MDT0000 complete [ 1501.158750] LustreError: 45288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.178829] LustreError: 45288:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1503.245813] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186718 with bad export cookie 594669371813251842 [ 1503.247877] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1503.261331] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1509.603663] Lustre: server umount lustre-MDT0001 complete [ 1521.976826] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1525.043913] Key type lgssc unregistered [ 1525.419859] LNet: 49028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.427927] LNetError: 49028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.443259] LNet: Removed LNI 192.168.201.109@tcp [ 1526.467794] Key type .llcrypt unregistered [ 1526.470116] Key type ._llcrypt unregistered [ 1553.756609] Key type ._llcrypt registered [ 1553.759140] Key type .llcrypt registered [ 1553.889098] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1555.110629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1555.339270] alg: No test for adler32 (adler32-zlib) [ 1556.545814] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 1556.845886] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1558.632184] Key type lgssc registered [ 1560.496802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.021966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1570.032994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.329480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.305672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1580.521830] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1580.553043] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1585.634410] LustreError: 50448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.509655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.848161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1593.294585] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1601.303296] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1607.578899] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1613.992093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.408898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1615.493730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1622.348798] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1630.843158] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1637.939549] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1641.508463] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1654.452386] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-08 7h01m07s [ 1656.392796] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1658.856684] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1658.874753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1658.875394] Lustre: Skipped 1 previous similar message [ 1658.897894] Lustre: Skipped 2 previous similar messages [ 1659.940215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1659.949917] Lustre: Skipped 3 previous similar messages [ 1664.886952] Lustre: server umount lustre-MDT0000 complete [ 1665.440423] LustreError: 50444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.357347] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1671.584914] LustreError: 50442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.603096] LustreError: 50442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1674.979345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.042058] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1675.214863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1675.261275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1676.268824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1679.685609] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1683.400626] Lustre: DEBUG MARKER: Wait for df (36740) ... [ 1684.975278] Lustre: DEBUG MARKER: done [ 1690.593566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.601213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.618212] Lustre: Skipped 1 previous similar message [ 1690.623641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.626529] Lustre: Skipped 1 previous similar message [ 1691.652296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.411032] Lustre: server umount lustre-OST0000 complete [ 1701.858224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.870169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.877581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.883208] Lustre: Skipped 1 previous similar message [ 1701.903933] Lustre: Skipped 3 previous similar messages [ 1706.365034] Lustre: server umount lustre-MDT0000 complete [ 1709.026315] LustreError: 51175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.033783] LustreError: 51175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1710.079680] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786186925 with bad export cookie 9917474255698218643 [ 1710.081181] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1710.089561] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1710.345030] Lustre: server umount lustre-MDT0001 complete [ 1721.299436] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1723.796632] Key type lgssc unregistered [ 1724.043989] LNet: 55386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.052611] LNetError: 55386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.078700] LNet: Removed LNI 192.168.201.109@tcp [ 1724.913463] Key type .llcrypt unregistered [ 1724.915591] Key type ._llcrypt unregistered [ 1764.310644] Key type ._llcrypt registered [ 1764.313764] Key type .llcrypt registered [ 1764.405882] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1765.775734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.834786] alg: No test for adler32 (adler32-zlib) [ 1766.871669] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 1767.159351] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1768.858420] Key type lgssc registered [ 1770.215990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.006311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.052034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.363821] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1780.380720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.492975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.769747] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1794.126832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.217301] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.280992] Lustre: 57595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1794.486840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1799.326543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1808.503556] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1815.826361] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1823.468900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1823.645435] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1823.904495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.120963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1831.318724] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1840.658636] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1848.759985] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-08 7h04m22s [ 1850.718605] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1859.029972] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1862.068526] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1862.150250] LustreError: 57601:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1882.624564] Lustre: lustre-MDT0000: Client f8074a9e-c567-4dd2-90a8-544b7ce90a26 (at 192.168.201.9@tcp) reconnecting [ 1902.216185] LustreError: 57601:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1904.986359] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1907.354788] Lustre: DEBUG MARKER: done [ 1917.923374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1917.936614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.952324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.433504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.450781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.476258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1918.485631] Lustre: Skipped 1 previous similar message [ 1919.674683] Lustre: server umount lustre-OST0000 complete [ 1927.653446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.658052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.671996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.162237] Lustre: server umount lustre-MDT0000 complete [ 1934.362567] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187150 with bad export cookie 409639120914705933 [ 1934.369531] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.371507] LustreError: 57589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1934.789091] Lustre: server umount lustre-MDT0001 complete [ 1947.694646] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1950.718926] Key type lgssc unregistered [ 1950.997185] LNet: 61937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.003317] LNetError: 61937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.014916] LNet: Removed LNI 192.168.201.109@tcp [ 1951.935545] Key type .llcrypt unregistered [ 1951.938539] Key type ._llcrypt unregistered [ 2002.093237] Key type ._llcrypt registered [ 2002.095666] Key type .llcrypt registered [ 2002.211725] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2004.179601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2004.389484] alg: No test for adler32 (adler32-zlib) [ 2005.630691] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 2005.985126] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2007.832276] Key type lgssc registered [ 2009.542913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2014.120145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2014.176151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.819884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2015.850369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2016.132514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.432483] Lustre: Failing over lustre-MDT0000 [ 2017.861100] Lustre: server umount lustre-MDT0000 complete [ 2026.750378] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:07:20 (1786187240) [ 2034.578034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.244445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.112457] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2051.283824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.484664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2051.550960] Lustre: 64734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.917181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.268212] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2067.956960] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2075.400854] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2083.191144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2083.517408] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2083.751585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.588360] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2092.217027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2099.810745] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2105.731828] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2113.000406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.003068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.017200] Lustre: Skipped 2 previous similar messages [ 2113.029070] Lustre: Skipped 1 previous similar message [ 2118.115841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.126790] Lustre: Skipped 1 previous similar message [ 2123.235751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.243837] Lustre: Skipped 4 previous similar messages [ 2127.328208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2127.497493] Lustre: server umount lustre-MDT0000 complete [ 2132.152752] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187347 with bad export cookie 2833692092701213190 [ 2132.158631] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.166950] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2132.479081] Lustre: server umount lustre-MDT0001 complete [ 2136.535965] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2142.168094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2149.864982] Lustre: 63659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187349/real 1786187349] req@000000003bea2c0f x1872953051491200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.907247] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.752248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.952274] Lustre: 63658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187353/real 1786187353] req@000000001391881b x1872953051491328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.976384] Lustre: 63657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187354/real 1786187354] req@000000005e702321 x1872953051491456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.089760] LustreError: 63655:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000de5a1944 x1872953051492096/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.572580] LustreError: 66584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.595621] LustreError: 66584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2157.721591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.892355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2158.690149] LustreError: 66585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.693332] Lustre: 63657:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187357/real 1786187357] req@0000000065ce729a x1872953051491712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2158.696362] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2159.719910] LustreError: 66584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.731603] LustreError: 66584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2162.777542] Lustre: 63658:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187362/real 1786187362] req@00000000a73ff712 x1872953051491840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2164.220528] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2165.221983] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.267874] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2170.352935] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.389713] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2175.197981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.080639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2181.119036] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2182.277741] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2191.471961] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2199.059319] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2211.816168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.822139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2211.831107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.843413] Lustre: Skipped 1 previous similar message [ 2213.421517] Lustre: server umount lustre-MDT0000 complete [ 2216.936208] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.962606] LustreError: 68505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2218.783589] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187434 with bad export cookie 2833692092701215465 [ 2218.788673] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2218.805096] LustreError: 64727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2219.485553] Lustre: server umount lustre-MDT0001 complete [ 2227.714694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2237.409134] Lustre: 63659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786187437/real 1786187437] req@0000000065439b9a x1872953051522304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786187453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2237.464292] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.216453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.618427] LustreError: 63655:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cdca2577 x1872953051523328/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2244.020902] LustreError: 66585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.123421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.199750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2250.367211] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2260.855520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.616941] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2268.113784] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2278.634717] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.696841] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2297.319688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.322613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2297.326260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.326269] Lustre: Skipped 2 previous similar messages [ 2297.351137] Lustre: Skipped 1 previous similar message [ 2301.208776] Lustre: server umount lustre-MDT0000 complete [ 2302.456979] LustreError: 72820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.492597] LustreError: 72820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 2306.061186] LustreError: 64726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187521 with bad export cookie 2833692092701216844 [ 2306.062735] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2306.074673] LustreError: 64726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2306.518209] Lustre: server umount lustre-MDT0001 complete [ 2313.899794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2317.975032] Lustre: DEBUG MARKER: files compared the same [ 2329.630881] Lustre: server umount lustre-OST0000 complete [ 2351.055740] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2354.192295] Key type lgssc unregistered [ 2354.657859] LNet: 74476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.668912] LNetError: 74476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.694762] LNet: Removed LNI 192.168.201.109@tcp [ 2357.106398] Key type .llcrypt unregistered [ 2357.112136] Key type ._llcrypt unregistered [ 2387.663596] Key type ._llcrypt registered [ 2387.669275] Key type .llcrypt registered [ 2387.774953] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2388.805849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.996292] alg: No test for adler32 (adler32-zlib) [ 2390.143475] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 2390.368131] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2392.056546] Key type lgssc registered [ 2393.253399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2403.239119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2403.287788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.131700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.982395] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2414.562478] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.600661] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2419.683313] LustreError: 75896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.922742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.391847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2429.480798] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2439.684599] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2447.043485] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2455.651426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2456.068953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.995500] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2472.453268] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2474.491181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2488.803982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2488.814905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.832046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2489.832634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.844846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2489.848362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2493.882288] Lustre: server umount lustre-OST0000 complete [ 2500.079560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.080429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.093114] Lustre: Skipped 1 previous similar message [ 2500.094227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2500.105314] Lustre: Skipped 2 previous similar messages [ 2504.697390] Lustre: server umount lustre-MDT0000 complete [ 2505.199302] LustreError: 75895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2509.464796] LustreError: 75877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187725 with bad export cookie 1543596590163137757 [ 2509.490855] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.807566] Lustre: server umount lustre-MDT0001 complete [ 2521.799192] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2524.857714] Key type lgssc unregistered [ 2525.136633] LNet: 79459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2525.140873] LNetError: 79459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.194303] LNet: Removed LNI 192.168.201.109@tcp [ 2527.811262] Key type .llcrypt unregistered [ 2527.815593] Key type ._llcrypt unregistered [ 2562.280248] Key type ._llcrypt registered [ 2562.281867] Key type .llcrypt registered [ 2562.495637] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2564.479438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.558701] alg: No test for adler32 (adler32-zlib) [ 2565.753931] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 2566.171816] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2568.016660] Key type lgssc registered [ 2569.800845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2580.132432] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2580.145959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2595.680721] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2600.801450] LustreError: 80854:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 2627.554890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.851483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2643.725873] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2654.646559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.951112] Lustre: *** cfs_fail_loc=706, val=0*** [ 2655.770976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.904816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2660.856803] LustreError: 80880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.891685] LustreError: 80880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2661.648479] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2661.859469] LustreError: 80879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.894380] LustreError: 80879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2666.979975] LustreError: 81940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.011805] LustreError: 81940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2671.909369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.100437] LustreError: 81941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.126791] LustreError: 81941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2672.505619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2677.583864] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2686.037461] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2693.611249] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2696.673807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2696.690545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.724381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2698.210723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2698.237461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.251231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2698.268409] Lustre: Skipped 1 previous similar message [ 2702.327267] Lustre: server umount lustre-OST0000 complete [ 2708.457680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.460976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.467928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.471457] Lustre: Skipped 1 previous similar message [ 2713.582432] LustreError: 81946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.615344] LustreError: 81946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2713.853089] Lustre: server umount lustre-MDT0000 complete [ 2718.813688] LustreError: 80861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786187934 with bad export cookie 14200536587478621688 [ 2718.824862] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.082834] Lustre: server umount lustre-MDT0001 complete [ 2734.092263] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2737.835342] Key type lgssc unregistered [ 2738.192984] LNet: 84475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2738.201917] LNetError: 84475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2738.216784] LNet: Removed LNI 192.168.201.109@tcp [ 2739.723945] Key type .llcrypt unregistered [ 2739.727201] Key type ._llcrypt unregistered [ 2772.803720] Key type ._llcrypt registered [ 2772.805247] Key type .llcrypt registered [ 2773.005850] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2774.591347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.756154] alg: No test for adler32 (adler32-zlib) [ 2776.130356] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 2776.424075] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2778.112151] Key type lgssc registered [ 2779.618347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2789.085345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2789.102640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.455044] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2804.862510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.619842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.860235] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.873862] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2811.557858] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2815.977402] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.008543] LustreError: 86580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2821.092701] LustreError: 86584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.122217] LustreError: 86584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2821.157151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2821.574896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.734542] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.764306] LustreError: 86581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2828.075147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2830.825861] LustreError: 86585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.848521] LustreError: 86585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2837.360985] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2837.394820] Lustre: MGS: Client 6d84cf42-7e87-4f88-ad98-56868b8d5f52 (at 0@lo) reconnecting [ 2837.435453] LustreError: 86595:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2837.746975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.836148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2842.691839] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2858.467887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.472732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.482843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.488481] Lustre: Skipped 1 previous similar message [ 2862.569980] Lustre: server umount lustre-OST0000 complete [ 2868.708584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.744601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2868.752871] Lustre: Skipped 2 previous similar messages [ 2873.703050] Lustre: server umount lustre-MDT0000 complete [ 2873.844370] LustreError: 88124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.867375] LustreError: 88124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2878.084918] Lustre: server umount lustre-MDT0001 complete [ 2882.308056] Lustre: server umount MGS complete [ 2894.811104] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:21:48 (1786188108) [ 2928.577381] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2938.682104] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2984.923587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.807035] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3002.100408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.340365] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3002.381843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3002.452532] Lustre: lustre-MDT0001: new disk, initializing [ 3002.532025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.562716] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3002.574255] Lustre: 93753:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3002.589371] LustreError: 93753:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3007.520234] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3019.860735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3020.180249] Lustre: lustre-OST0000: new disk, initializing [ 3020.183715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3020.192285] Lustre: 94519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3020.285458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.165902] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3031.201861] Lustre: 94960:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ee5b2abe x1872953859940352/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786188262 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3038.658416] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3044.910198] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.917315] Lustre: MGS: Client adc231a8-c54e-40eb-85d1-cc5c0c8dd8c7 (at 0@lo) reconnecting [ 3044.947564] Lustre: 93021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3044.959286] LustreError: 93739:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3044.985177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3045.041996] Lustre: lustre-MDT0000: new disk, initializing [ 3045.170316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.193726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3045.268770] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3046.246054] Lustre: 95588:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000ff3cdeb6 x1872953859947008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786188277 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3050.457280] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3051.303412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3051.309691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3051.406958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3077.089790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.089943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.102376] LustreError: Skipped 1 previous similar message [ 3077.103717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.117406] Lustre: Skipped 2 previous similar messages [ 3077.137681] Lustre: Skipped 1 previous similar message [ 3082.215202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3082.229139] Lustre: Skipped 1 previous similar message [ 3082.706881] Lustre: server umount lustre-OST0000 complete [ 3092.451551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.452990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3092.483143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3092.521891] Lustre: Skipped 1 previous similar message [ 3094.365416] Lustre: server umount lustre-MDT0000 complete [ 3097.584451] LustreError: 93725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.598351] LustreError: 93725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3105.052807] Lustre: server umount lustre-MDT0001 complete [ 3109.915140] Lustre: server umount MGS complete [ 3124.023878] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:25:38 (1786188338) [ 3148.037268] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3151.554543] Key type lgssc unregistered [ 3151.992912] LNet: 98091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.002134] LNetError: 98091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.019391] LNet: Removed LNI 192.168.201.109@tcp [ 3153.344933] Key type .llcrypt unregistered [ 3153.347957] Key type ._llcrypt unregistered [ 3172.660475] Key type ._llcrypt registered [ 3172.662094] Key type .llcrypt registered [ 3172.741864] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3173.630899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.667225] alg: No test for adler32 (adler32-zlib) [ 3174.865365] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3175.125668] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3176.792204] Key type lgssc registered [ 3177.887642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3188.625351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3188.661371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.736718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.571917] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3201.012362] LustreError: 99318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3201.034579] LustreError: 99318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3206.115945] LustreError: 99317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.563154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.922810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.780664] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3220.564342] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.955895] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3235.472497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3235.826182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.970199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3243.473563] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3252.980788] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.458994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.467582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.485719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3263.531585] Lustre: Skipped 1 previous similar message [ 3268.581393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.604634] Lustre: Skipped 2 previous similar messages [ 3268.855704] Lustre: server umount lustre-OST0000 complete [ 3273.702445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.705451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3273.728543] Lustre: Skipped 1 previous similar message [ 3273.745318] Lustre: Skipped 1 previous similar message [ 3278.820665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.825380] Lustre: Skipped 1 previous similar message [ 3279.736535] Lustre: server umount lustre-MDT0000 complete [ 3283.940799] LustreError: 99317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.955601] LustreError: 99317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3284.569711] LustreError: 99297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188500 with bad export cookie 5301931638640570484 [ 3284.575900] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3284.579979] LustreError: 99297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3284.877630] Lustre: server umount lustre-MDT0001 complete [ 3299.037221] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3303.074873] Key type lgssc unregistered [ 3303.488043] LNet: 102854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3303.504046] LNetError: 102854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3303.553072] LNet: Removed LNI 192.168.201.109@tcp [ 3304.803807] Key type .llcrypt unregistered [ 3304.805917] Key type ._llcrypt unregistered [ 3324.431563] Key type ._llcrypt registered [ 3324.433854] Key type .llcrypt registered [ 3324.579542] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3326.115464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3326.136204] alg: No test for adler32 (adler32-zlib) [ 3327.279126] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3327.835141] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3329.624163] Key type lgssc registered [ 3331.164174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3338.871705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3338.882746] LustreError: 103942:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3338.886568] LustreError: 103943:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3338.893445] LustreError: 103942:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3338.904985] LustreError: 103942:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3338.906987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.912379] LustreError: 103942:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3340.472838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.905423] LustreError: 103979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3345.948854] LustreError: 103979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3349.770539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.023748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.051820] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3361.418829] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lsmod [ 3365.032140] LustreError: 105148:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3365.038679] LustreError: 105147:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3365.063039] LustreError: 105148:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3365.069982] LustreError: 105148:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3365.078556] LustreError: 105148:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3365.130440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.808973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.860692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3370.466579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3370.473733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3370.496779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.002664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.003895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3372.031253] Lustre: Skipped 1 previous similar message [ 3372.045684] Lustre: Skipped 1 previous similar message [ 3375.074910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.083900] Lustre: Skipped 3 previous similar messages [ 3375.753655] Lustre: server umount lustre-MDT0000 complete [ 3376.106250] LustreError: 103979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3376.127666] LustreError: 103979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3380.147939] LustreError: 103954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188595 with bad export cookie 15960212126696686075 [ 3380.200259] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3380.256166] LustreError: 104411:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000033c1fa9d x1872954437687424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3380.677590] Lustre: server umount lustre-MDT0001 complete [ 3396.020688] Lustre: server umount lustre-OST0000 complete [ 3406.521100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.243678] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.372988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.456807] LustreError: 106212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.320232] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3416.551294] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.796512] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.833598] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3428.882415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.260713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3434.584765] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3444.311411] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3452.079758] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3460.253045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.814769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3470.341815] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3478.313480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3481.558557] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3489.768959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3489.772490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3489.793973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.857323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.863920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.880355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3495.641648] Lustre: server umount lustre-OST0000 complete [ 3501.025836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3501.030347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.101682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3504.112177] Lustre: Skipped 3 previous similar messages [ 3514.337961] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3514.755101] Lustre: server umount lustre-MDT0000 complete [ 3518.433498] LustreError: 107058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3518.452430] LustreError: 107058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3518.801616] LustreError: 106192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188734 with bad export cookie 15960212126696687006 [ 3518.804480] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.819017] LustreError: 106192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3519.421425] Lustre: server umount lustre-MDT0001 complete [ 3533.346627] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3536.899336] Key type lgssc unregistered [ 3537.167510] LNet: 109757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.173857] LNetError: 109757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3537.191459] LNet: Removed LNI 192.168.201.109@tcp [ 3538.204628] Key type .llcrypt unregistered [ 3538.206435] Key type ._llcrypt unregistered [ 3567.303895] Key type ._llcrypt registered [ 3567.306893] Key type .llcrypt registered [ 3567.451374] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3568.879855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3568.907072] alg: No test for adler32 (adler32-zlib) [ 3569.917715] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3570.130667] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3571.824410] Key type lgssc registered [ 3573.300536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3582.888557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3582.911570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.470783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.436576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3594.724328] LustreError: 111176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.739915] LustreError: 111176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3599.847332] LustreError: 111177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3599.979397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.379534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3605.935542] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3615.240634] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3622.246365] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3630.289835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3630.625157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.280117] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3645.790699] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3655.215269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3662.010063] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3667.215342] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3675.636857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.637204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3675.649600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.654017] Lustre: Skipped 1 previous similar message [ 3675.664190] LustreError: Skipped 1 previous similar message [ 3677.789191] Lustre: server umount lustre-OST0000 complete [ 3685.863407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.866900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.891982] Lustre: Skipped 1 previous similar message [ 3685.898634] Lustre: Skipped 1 previous similar message [ 3688.743950] Lustre: server umount lustre-MDT0000 complete [ 3690.980128] LustreError: 111173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.004485] LustreError: 111173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3693.418345] LustreError: 111158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786188909 with bad export cookie 3581344550415693813 [ 3693.439064] LustreError: 111158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3693.442407] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3693.944956] Lustre: server umount lustre-MDT0001 complete [ 3707.246504] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3710.273929] Key type lgssc unregistered [ 3710.580293] LNet: 114842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3710.583258] LNetError: 114842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3710.627046] LNet: Removed LNI 192.168.201.109@tcp [ 3711.524820] Key type .llcrypt unregistered [ 3711.526960] Key type ._llcrypt unregistered [ 3732.338275] Key type ._llcrypt registered [ 3732.344391] Key type .llcrypt registered [ 3732.547752] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3734.318814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3734.363449] alg: No test for adler32 (adler32-zlib) [ 3735.628334] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3736.071891] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3737.800173] Key type lgssc registered [ 3739.155944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3749.082643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3749.127317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.807264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.088313] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3761.130325] LustreError: 116069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.146638] LustreError: 116069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3765.546029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.124951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3770.807287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3779.285678] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3786.608990] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3795.228716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.707662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.725834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3803.554992] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3812.183988] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3822.562262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.583100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.608563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3826.660544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.662434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3826.669540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.704280] Lustre: Skipped 1 previous similar message [ 3827.929557] Lustre: server umount lustre-OST0000 complete [ 3832.805492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3832.831105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.871764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.901914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.909924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3836.915295] Lustre: Skipped 1 previous similar message [ 3838.990093] Lustre: server umount lustre-MDT0000 complete [ 3842.017724] LustreError: 116070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.051626] LustreError: 116070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3843.429076] LustreError: 116049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189059 with bad export cookie 17432852817814896133 [ 3843.437088] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.440983] LustreError: 116049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3843.864552] Lustre: server umount lustre-MDT0001 complete [ 3861.172605] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3865.258892] Key type lgssc unregistered [ 3865.740378] LNet: 119608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3865.765313] LNetError: 119608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3865.791030] LNet: Removed LNI 192.168.201.109@tcp [ 3867.218561] Key type .llcrypt unregistered [ 3867.220259] Key type ._llcrypt unregistered [ 3894.479806] Key type ._llcrypt registered [ 3894.482634] Key type .llcrypt registered [ 3894.594620] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3896.047496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.433542] alg: No test for adler32 (adler32-zlib) [ 3897.640695] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3897.952842] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3899.696190] Key type lgssc registered [ 3901.251332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.986368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3910.011425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.485405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.962888] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3921.898562] LustreError: 121025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.929718] LustreError: 121025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3924.254805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.629646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.095929] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3939.640912] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3946.789987] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3955.520147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3955.999488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.045217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3963.665266] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3972.366956] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3980.545381] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3984.684338] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3984.700448] Lustre: Skipped 2 previous similar messages [ 3987.045699] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3987.091842] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3989.294606] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3989.299357] Lustre: Skipped 2 previous similar messages [ 3996.283944] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3996.295063] Lustre: Skipped 1 previous similar message [ 4002.043133] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4002.056966] Lustre: Skipped 2 previous similar messages [ 4003.802838] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp [ 4009.153926] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.9/24@tcp in log lustre-MDT0000 [ 4009.159973] Lustre: Skipped 2 previous similar messages [ 4009.352254] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 4009.355882] Lustre: Skipped 1 previous similar message [ 4016.609977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4016.618235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.620361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4016.640111] Lustre: Skipped 1 previous similar message [ 4021.731509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.748366] Lustre: Skipped 2 previous similar messages [ 4022.327564] Lustre: server umount lustre-OST0000 complete [ 4031.977790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.988068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4031.989841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.992477] Lustre: Skipped 1 previous similar message [ 4033.371124] Lustre: server umount lustre-MDT0000 complete [ 4037.090553] LustreError: 121026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.100350] LustreError: 121026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4037.522404] LustreError: 121006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189253 with bad export cookie 6368293507492531122 [ 4037.547958] LustreError: 121006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4037.548496] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4037.869612] Lustre: server umount lustre-MDT0001 complete [ 4049.783583] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4053.181694] Key type lgssc unregistered [ 4053.645756] LNet: 125830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4053.666543] LNetError: 125830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4053.690968] LNet: Removed LNI 192.168.201.109@tcp [ 4054.717476] Key type .llcrypt unregistered [ 4054.720044] Key type ._llcrypt unregistered [ 4088.077693] Key type ._llcrypt registered [ 4088.080254] Key type .llcrypt registered [ 4088.203471] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4090.173492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.735704] alg: No test for adler32 (adler32-zlib) [ 4091.839420] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 4092.164159] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4093.840663] Key type lgssc registered [ 4095.630925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4104.604617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4104.626808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.244265] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4106.267583] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.9/24@tcp [ 4106.271871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.602396] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4115.428115] LustreError: 127396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.469759] LustreError: 127396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4120.554581] LustreError: 127397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.669274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.062776] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4121.089687] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 4121.166959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4125.946829] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4135.042472] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4142.069774] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.076972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.323814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4161.048914] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4172.880816] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.933072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4191.201560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4191.207463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.215877] LustreError: Skipped 1 previous similar message [ 4191.223348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4191.242555] Lustre: Skipped 1 previous similar message [ 4191.258371] Lustre: Skipped 1 previous similar message [ 4195.401459] Lustre: server umount lustre-OST0000 complete [ 4201.443126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.478448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.484325] Lustre: Skipped 1 previous similar message [ 4201.513115] Lustre: Skipped 1 previous similar message [ 4206.578281] LustreError: 128396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.593767] LustreError: 128396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4206.761494] Lustre: server umount lustre-MDT0000 complete [ 4210.658443] LustreError: 127393:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.706746] LustreError: 127393:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4213.625746] LustreError: 127378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189429 with bad export cookie 16383253988179697244 [ 4213.627996] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4213.638563] LustreError: 127378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4214.438864] Lustre: server umount lustre-MDT0001 complete [ 4229.396594] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4232.299912] Key type lgssc unregistered [ 4232.900555] LNet: 130991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4232.915714] LNetError: 130991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4233.969493] LNet: Removed LNI 192.168.201.109@tcp [ 4235.773518] Key type .llcrypt unregistered [ 4235.776758] Key type ._llcrypt unregistered [ 4269.798582] Key type ._llcrypt registered [ 4269.803392] Key type .llcrypt registered [ 4269.974228] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4289.189844] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4290.715304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4290.812562] alg: No test for adler32 (adler32-zlib) [ 4292.009966] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 4292.345143] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4294.104266] Key type lgssc registered [ 4295.529117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4341.148175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4341.219288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.743601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4342.846542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4342.999750] Lustre: lustre-MDT0000: new disk, initializing [ 4343.162067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.225465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4350.932842] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4364.479922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.648371] Lustre: 134393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4364.734626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4364.746039] Lustre: Skipped 1 previous similar message [ 4364.848747] Lustre: lustre-MDT0001: new disk, initializing [ 4364.934769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.979776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4364.992347] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4370.966962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4384.342464] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4392.614315] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4402.486905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4402.955508] Lustre: lustre-OST0000: new disk, initializing [ 4402.962665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4402.972107] Lustre: 136366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4403.080703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.125173] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4412.074413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4412.097306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4412.269418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4421.127419] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4427.421989] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4427.775088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4434.357111] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4434.752461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4437.986051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.993271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4437.995323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4438.022666] Lustre: Skipped 1 previous similar message [ 4438.063473] Lustre: Skipped 1 previous similar message [ 4443.118862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.231779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.245874] Lustre: Skipped 2 previous similar messages [ 4451.296934] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.523247] Lustre: server umount lustre-OST0000 complete [ 4457.441854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.447939] LustreError: Skipped 1 previous similar message [ 4457.452362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.467181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.465748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.746318] Lustre: server umount lustre-MDT0000 complete [ 4467.072913] LustreError: 134385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189682 with bad export cookie 4678776781669426192 [ 4467.078587] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.088767] LustreError: 134385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4467.821917] Lustre: server umount lustre-MDT0001 complete [ 4476.146546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.865695] LustreError: 138585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.888862] LustreError: 138585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4477.032595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.591510] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4482.032203] LustreError: 138586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.069073] LustreError: 138586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4487.138178] LustreError: 138585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.258020] LustreError: 138586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.942180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.588650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.037511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4510.805936] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.623307] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4526.617319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4527.364600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.591852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4545.537327] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4551.614378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4551.955165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4557.527539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4557.828746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4570.582669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.943880] Lustre: lustre-OST0001: new disk, initializing [ 4570.957339] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4570.980437] Lustre: 141957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.166213] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4580.052888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4580.066606] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4580.095821] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4580.282323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4592.162810] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4592.390323] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4599.399507] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4599.775415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4607.552846] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89cdc3164000.ost_server_uuid 50 [ 4609.464747] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89cdc3164000.ost_server_uuid in IDLE state after 0 sec [ 4621.286166] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.287520] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.294118] LustreError: Skipped 1 previous similar message [ 4621.294995] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4621.295030] Lustre: Skipped 2 previous similar messages [ 4621.327483] Lustre: Skipped 1 previous similar message [ 4631.526524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4631.537386] Lustre: Skipped 4 previous similar messages [ 4632.032149] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4632.208315] Lustre: server umount lustre-OST0001 complete [ 4636.645944] LustreError: 140410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.689080] LustreError: 140410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4639.202170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.215692] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4644.073722] Lustre: server umount lustre-OST0000 complete [ 4652.007524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.027570] Lustre: Skipped 1 previous similar message [ 4652.047353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4652.055994] Lustre: Skipped 4 previous similar messages [ 4664.288321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4664.669986] Lustre: server umount lustre-MDT0000 complete [ 4667.366326] LustreError: 138585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.417967] LustreError: 138585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4669.890289] LustreError: 138567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786189885 with bad export cookie 4678776781669427074 [ 4669.914397] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.498227] Lustre: server umount lustre-MDT0001 complete [ 4698.651600] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4701.941351] Key type lgssc unregistered [ 4702.314749] LNet: 144968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4702.329398] LNetError: 144968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4703.400434] LNet: Removed LNI 192.168.201.109@tcp [ 4704.741425] Key type .llcrypt unregistered [ 4704.743117] Key type ._llcrypt unregistered [ 4754.975097] Key type ._llcrypt registered [ 4754.977379] Key type .llcrypt registered [ 4755.119668] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4774.218707] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4776.232437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4776.285773] alg: No test for adler32 (adler32-zlib) [ 4777.533589] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 4777.959537] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4779.762547] Key type lgssc registered [ 4781.318488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4828.107198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4828.159857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.867694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4829.925215] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4830.147388] Lustre: lustre-MDT0000: new disk, initializing [ 4830.287825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4830.314586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4836.433544] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4849.719892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.936366] Lustre: 149131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4850.032804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4850.036379] Lustre: Skipped 1 previous similar message [ 4850.117659] Lustre: lustre-MDT0001: new disk, initializing [ 4850.199971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4850.220948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4850.228251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4856.064322] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4866.797074] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4874.045358] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4883.252438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4883.571425] Lustre: lustre-OST0000: new disk, initializing [ 4883.574887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4883.580337] Lustre: 151101:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4883.707732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.607234] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4891.728292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4891.738563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4891.880426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4903.037700] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.264197] Lustre: Failing over lustre-OST0000 [ 4909.511861] Lustre: server umount lustre-OST0000 complete [ 4926.944178] Lustre: 147353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190126/real 1786190126] req@00000000a94f6a53 x1872955958135936/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786190142 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4926.994499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4928.993346] Lustre: 147354:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190128/real 1786190128] req@00000000b2304d99 x1872955958136320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786190144 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4929.022706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.080761] Lustre: 147351:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190128/real 1786190128] req@0000000017d38b1c x1872955958137088/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786190144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4930.143535] Lustre: 147351:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4932.098723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.726886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.779192] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4934.112117] Lustre: 147353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190133/real 1786190133] req@00000000e8ccde59 x1872955958137984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786190149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4934.122337] Lustre: 147353:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4936.658759] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4938.849098] Lustre: 147352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190138/real 1786190138] req@00000000fc850009 x1872955958139264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786190154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4938.903911] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4938.906488] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4940.732880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4944.892179] Lustre: Failing over lustre-MDT0000 [ 4945.656574] Lustre: server umount lustre-MDT0000 complete [ 4946.915879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4946.937465] Lustre: Skipped 1 previous similar message [ 4951.979260] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.039020] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4957.089475] LustreError: 149137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.224512] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.245401] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4963.301428] Lustre: 147352:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786190162/real 1786190162] req@000000007759868c x1872955958152192/t0(0) o400->MGC192.168.201.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786190178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4963.346665] Lustre: 147352:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4963.364920] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.139877] LustreError: 149138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.168431] LustreError: 149138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4967.522290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.259705] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.284931] LustreError: 150357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4973.844234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.931736] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4977.577220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4979.178935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4979.196757] Lustre: Skipped 1 previous similar message [ 4979.233200] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4979.299573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4980.032434] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4990.437667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4990.452681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.467401] Lustre: Skipped 1 previous similar message [ 4990.475945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4994.542157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4994.542591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.657590] Lustre: server umount lustre-OST0000 complete [ 5004.770495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.773053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5004.787305] Lustre: Skipped 1 previous similar message [ 5004.798579] Lustre: Skipped 2 previous similar messages [ 5009.889698] LustreError: 149142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.937105] LustreError: 149142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5010.714759] Lustre: server umount lustre-MDT0000 complete [ 5015.031850] LustreError: 149123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190230 with bad export cookie 6113199028833747858 [ 5015.045294] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.050120] LustreError: 149123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5015.535553] Lustre: server umount lustre-MDT0001 complete [ 5028.994616] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5032.726210] Key type lgssc unregistered [ 5033.136825] LNet: 154764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5033.147079] LNetError: 154764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5033.167197] LNet: Removed LNI 192.168.201.109@tcp [ 5034.069709] Key type .llcrypt unregistered [ 5034.071354] Key type ._llcrypt unregistered [ 5073.017531] Key type ._llcrypt registered [ 5073.022737] Key type .llcrypt registered [ 5073.162107] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5074.718298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5074.923148] alg: No test for adler32 (adler32-zlib) [ 5076.285812] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 5076.580226] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5078.328202] Key type lgssc registered [ 5080.455234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5090.103721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5090.136421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.933511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.696743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5102.053019] LustreError: 156287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5102.073946] LustreError: 156287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5107.184172] LustreError: 156288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.226810] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5112.294215] LustreError: 156287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.733440] LustreError: 156316:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5112.748577] LustreError: 156316:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5112.755669] LustreError: 156316:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5113.305865] Lustre: server umount lustre-MDT0000 complete [ 5131.191574] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5141.924696] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5186.524105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.965241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5187.022701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5187.191466] Lustre: lustre-MDT0000: new disk, initializing [ 5187.316483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.343422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5193.420879] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5206.743360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.955353] Lustre: 159621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5207.020217] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5207.032237] Lustre: Skipped 1 previous similar message [ 5207.173155] Lustre: lustre-MDT0001: new disk, initializing [ 5207.302170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5207.346479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5207.366401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5212.743200] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5225.263496] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5233.414354] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5242.069551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.577747] Lustre: lustre-OST0000: new disk, initializing [ 5242.584851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5242.623267] Lustre: 161590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5242.757644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.956372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5250.669675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5250.674853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5250.775856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.028535] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5266.253828] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5268.261548] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:01:22 (1786190482) [ 5512.162822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.163391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.200761] Lustre: Skipped 2 previous similar messages [ 5512.214713] Lustre: Skipped 1 previous similar message [ 5516.860235] Lustre: server umount lustre-MDT0000 complete [ 5517.283174] LustreError: 159629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.305379] LustreError: 159629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5521.753160] LustreError: 159615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190737 with bad export cookie 12794505212751709990 [ 5521.767238] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.772260] LustreError: 159615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5522.400866] LustreError: 159627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5522.404167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5522.434071] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5526.433134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5531.554983] LustreError: 159628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.555236] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5531.582794] LustreError: 159628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5536.227249] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5536.777596] Lustre: server umount lustre-MDT0001 complete [ 5547.453550] Lustre: server umount lustre-OST0000 complete [ 5560.851322] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5574.344792] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5615.957456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.239785] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.294058] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5616.376753] Lustre: lustre-MDT0000: new disk, initializing [ 5616.432830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.443551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5622.510444] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5634.605481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.835319] Lustre: 166264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5634.935145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5634.941072] Lustre: Skipped 1 previous similar message [ 5635.091849] Lustre: lustre-MDT0001: new disk, initializing [ 5635.221311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5635.281032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5635.295924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5640.210790] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5650.262984] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.471887] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5663.818948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5664.130682] Lustre: lustre-OST0000: new disk, initializing [ 5664.134344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5664.142557] Lustre: 168234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5664.244682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.528292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5671.540818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5671.709794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5671.767889] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5682.402208] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5688.519605] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5688.751926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5694.052031] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5694.316663] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5697.512033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5697.518166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5697.537978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.553902] Lustre: Skipped 1 previous similar message [ 5702.634132] LustreError: 168581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.651946] LustreError: 168581:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5702.850657] Lustre: server umount lustre-OST0000 complete [ 5707.236095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.247196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.262228] Lustre: Skipped 1 previous similar message [ 5707.269113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.351135] Lustre: server umount lustre-MDT0000 complete [ 5718.272205] LustreError: 166258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786190933 with bad export cookie 12794505212751868820 [ 5718.275200] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5718.286237] LustreError: 166258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5718.987696] Lustre: server umount lustre-MDT0001 complete [ 5730.685915] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:09:04 (1786190944) [ 5751.518049] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5763.870825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5803.571902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.852802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5803.893898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5804.053820] Lustre: lustre-MDT0000: new disk, initializing [ 5804.206302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.242886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5809.488342] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5820.240021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.347465] Lustre: 172992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5820.664994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5825.072363] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5835.215985] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5841.187059] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5847.706402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5847.924409] Lustre: lustre-OST0000: new disk, initializing [ 5847.927104] Lustre: Skipped 1 previous similar message [ 5847.935319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5847.941459] Lustre: Skipped 2 previous similar messages [ 5847.947452] Lustre: 174971:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5853.778333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5853.788123] Lustre: Skipped 1 previous similar message [ 5853.793683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5854.003689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5854.217647] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5863.459596] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.669172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.671212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.675593] LustreError: Skipped 1 previous similar message [ 5874.685537] Lustre: Skipped 1 previous similar message [ 5874.689218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.689227] Lustre: Skipped 3 previous similar messages [ 5884.384146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5884.533503] Lustre: server umount lustre-OST0000 complete [ 5890.018060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5890.044546] Lustre: Skipped 2 previous similar messages [ 5894.381485] Lustre: server umount lustre-MDT0000 complete [ 5895.140280] LustreError: 173005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.162389] LustreError: 173005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5898.363559] LustreError: 172985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191114 with bad export cookie 12794505212751869702 [ 5898.370350] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5898.374252] LustreError: 172985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5898.630561] Lustre: server umount lustre-MDT0001 complete [ 5907.254577] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:12:01 (1786191121) [ 5924.554529] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5934.924124] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5969.016642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.234562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5969.259319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5969.344995] Lustre: lustre-MDT0000: new disk, initializing [ 5969.459930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.472619] Lustre: Skipped 2 previous similar messages [ 5969.484513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5974.523557] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5984.863540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.984619] Lustre: 179296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5985.268462] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5990.010579] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5999.077971] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6005.352834] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6012.317189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6012.578447] Lustre: 181274:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6018.083806] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6020.681167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6020.765240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6027.720280] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6036.453800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6036.455972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.463020] LustreError: Skipped 1 previous similar message [ 6036.464254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6036.464260] Lustre: Skipped 5 previous similar messages [ 6036.499907] Lustre: Skipped 1 previous similar message [ 6040.568504] Lustre: server umount lustre-OST0000 complete [ 6046.689368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6050.813899] Lustre: server umount lustre-MDT0000 complete [ 6051.810518] LustreError: 181944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.821088] LustreError: 181944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6053.963093] LustreError: 179290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191269 with bad export cookie 12794505212751870787 [ 6053.969063] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.972845] LustreError: 179290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6054.419691] Lustre: server umount lustre-MDT0001 complete [ 6064.254836] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6067.162606] Key type lgssc unregistered [ 6067.412689] LNet: 183132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6067.422735] LNetError: 183132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6067.438969] LNet: Removed LNI 192.168.201.109@tcp [ 6068.144658] Key type .llcrypt unregistered [ 6068.146787] Key type ._llcrypt unregistered [ 6090.060132] Key type ._llcrypt registered [ 6090.062749] Key type .llcrypt registered [ 6090.178978] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6091.361906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.417093] alg: No test for adler32 (adler32-zlib) [ 6092.482309] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 6092.694547] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6094.376904] Key type lgssc registered [ 6095.457807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6102.734817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6102.750275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.094633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.349541] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6114.274763] LustreError: 184547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6114.286935] LustreError: 184547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6116.483075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.936497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6120.766709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6127.981428] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6132.897329] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6139.040451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6139.342906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.104303] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6148.601048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6151.871409] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6172.133688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6172.142517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6172.155786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6174.178689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6174.189056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6174.198837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.218629] Lustre: Skipped 1 previous similar message [ 6176.257555] Lustre: server umount lustre-OST0000 complete [ 6183.396618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6183.403883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6183.417659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6185.695196] Lustre: server umount lustre-MDT0000 complete [ 6188.771306] LustreError: 184528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191404 with bad export cookie 13117097765200346009 [ 6188.774582] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.781887] LustreError: 184528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6189.092244] Lustre: server umount lustre-MDT0001 complete [ 6198.647871] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6200.834410] Key type lgssc unregistered [ 6201.044635] LNet: 188107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6201.047419] LNetError: 188107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6201.065563] LNet: Removed LNI 192.168.201.109@tcp [ 6201.889216] Key type .llcrypt unregistered [ 6201.891085] Key type ._llcrypt unregistered [ 6224.912385] Key type ._llcrypt registered [ 6224.914620] Key type .llcrypt registered [ 6225.035840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6226.086772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6226.236551] alg: No test for adler32 (adler32-zlib) [ 6227.280919] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 6227.488524] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6229.152267] Key type lgssc registered [ 6230.587459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.929232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.943645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6239.519316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.496684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6249.955266] LustreError: 189524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6249.974831] LustreError: 189524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6251.024755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.378304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6255.047848] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6261.233712] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6265.989405] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6271.272418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6271.550026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.465184] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6282.409063] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6284.853172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6293.440028] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6295.009488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6295.015760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.027958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6295.033225] Lustre: Skipped 1 previous similar message [ 6295.041823] Lustre: Skipped 1 previous similar message [ 6298.536595] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6303.657225] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6303.662877] Lustre: Skipped 2 previous similar messages [ 6306.272305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6306.359646] Lustre: server umount lustre-OST0000 complete [ 6312.229420] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89cdd1cbf000.ost_server_uuid 50 [ 6313.444946] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6328.289791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6328.294776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.319968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.322375] Lustre: Skipped 2 previous similar messages [ 6330.851115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.426619] Lustre: server umount lustre-MDT0000 complete [ 6335.596145] LustreError: 189503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191551 with bad export cookie 5045134150021856843 [ 6335.597875] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6335.601730] LustreError: 189503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6335.861752] Lustre: server umount lustre-MDT0001 complete [ 6343.231603] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:19:17 (1786191557) [ 6348.261661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6348.603217] LustreError: 193438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.686410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.262977] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6353.897241] LustreError: 193439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.901655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6359.011150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6359.015461] Lustre: Skipped 2 previous similar messages [ 6359.135936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6363.242653] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6369.623433] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6374.656897] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6380.535587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6380.814591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.501362] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6392.408655] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6395.204226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6397.429690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.538565] Lustre: lustre-OST0001: new disk, initializing [ 6397.542371] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6397.547946] Lustre: 196278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6397.622473] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6402.316830] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6405.197507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6405.202412] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6405.272210] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6409.809971] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.817897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.825956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.833608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6426.085981] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6426.202787] Lustre: server umount lustre-OST0000 complete [ 6430.523533] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6430.647584] LustreError: 195260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.733582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6433.956349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6434.144504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6435.745135] LustreError: 195260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6435.760133] LustreError: 195260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6446.561730] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6446.564094] LustreError: 195261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.568052] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.568061] Lustre: Skipped 1 previous similar message [ 6446.575223] LustreError: Skipped 2 previous similar messages [ 6446.615890] LustreError: 195261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6451.683247] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6451.686790] Lustre: Skipped 11 previous similar messages [ 6452.340187] Lustre: server umount lustre-OST0001 complete [ 6456.298212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6456.305057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6456.319540] Lustre: Skipped 1 previous similar message [ 6461.244378] Lustre: server umount lustre-MDT0000 complete [ 6465.345595] LustreError: 193418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191680 with bad export cookie 5045134150021857921 [ 6465.353823] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.354613] LustreError: 193418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6465.748658] Lustre: server umount lustre-MDT0001 complete [ 6484.642640] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:21:39 (1786191699) [ 6489.590484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.830060] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6489.851449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6490.044173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.901331] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6501.107191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.191694] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6501.231648] Lustre: 199715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6505.560635] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6511.248923] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6515.679590] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6520.716909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6520.900498] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6527.085699] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6528.088710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6533.446223] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6538.650227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.697856] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6543.685643] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6549.126944] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6553.569273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.570881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.578956] LustreError: Skipped 1 previous similar message [ 6553.586888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.587611] Lustre: Skipped 2 previous similar messages [ 6553.602026] Lustre: Skipped 3 previous similar messages [ 6565.344254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6565.462198] Lustre: server umount lustre-OST0000 complete [ 6569.897648] LustreError: 201564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.922755] LustreError: 201564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6580.195866] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6585.020408] Lustre: server umount lustre-OST0001 complete [ 6588.385780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.390257] Lustre: Skipped 2 previous similar messages [ 6594.292683] Lustre: server umount lustre-MDT0000 complete [ 6597.476815] LustreError: 199705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191813 with bad export cookie 5045134150021859328 [ 6597.491764] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6597.493434] LustreError: 199705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6597.760567] Lustre: server umount lustre-MDT0001 complete [ 6616.105729] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:23:50 (1786191830) [ 6629.643490] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6636.705598] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6661.242830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.429713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6661.445532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6661.501246] Lustre: lustre-MDT0000: new disk, initializing [ 6661.547930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6661.558092] Lustre: Skipped 3 previous similar messages [ 6661.566922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6664.121483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6671.179835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.247913] Lustre: 207804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6671.278750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6671.282751] Lustre: Skipped 1 previous similar message [ 6671.326404] Lustre: lustre-MDT0001: new disk, initializing [ 6671.402959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6671.408668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6673.930268] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6680.424789] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6684.159949] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6688.811458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6689.058641] Lustre: lustre-OST0000: new disk, initializing [ 6689.062301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6689.066477] Lustre: 209771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6693.260590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6697.493121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6697.500424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6697.565590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6699.102214] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6701.746451] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6701.858367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6704.363634] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6704.482194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6707.686204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6707.686846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.695448] LustreError: Skipped 3 previous similar messages [ 6707.701517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6707.716949] Lustre: Skipped 2 previous similar messages [ 6707.717760] Lustre: Skipped 15 previous similar messages [ 6711.696645] Lustre: server umount lustre-OST0000 complete [ 6720.274387] Lustre: server umount lustre-MDT0000 complete [ 6722.578703] LustreError: 207797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786191938 with bad export cookie 5045134150021860896 [ 6722.579357] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6722.585374] LustreError: 207797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6722.769665] Lustre: server umount lustre-MDT0001 complete [ 6727.714858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6728.002140] LustreError: 211984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.015649] LustreError: 211984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6730.797550] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6736.829804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.940287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6744.739743] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6748.215440] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.174753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.093614] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6760.592980] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6763.139393] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6763.247193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6765.643826] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6765.767429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.870146] Lustre: server umount lustre-OST0000 complete [ 6772.820606] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6775.869554] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6802.190186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6802.403275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.408702] Lustre: Skipped 5 previous similar messages [ 6806.213692] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6806.500765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.519784] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6806.522587] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6806.535979] Lustre: Skipped 1 previous similar message [ 6810.944193] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6822.774683] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 6826.982247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6826.991034] LustreError: Skipped 3 previous similar messages [ 6827.242119] Lustre: server umount lustre-OST0000 complete [ 6829.628519] Lustre: server umount lustre-MDT0000 complete [ 6831.928081] LustreError: 211965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192047 with bad export cookie 5045134150021861771 [ 6831.934735] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.215616] Lustre: server umount lustre-MDT0001 complete [ 6843.733892] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:27:38 (1786192058) [ 6848.020132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.299783] LustreError: 217964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.316371] LustreError: 217964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6851.460308] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6857.573779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.852386] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6866.050895] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6869.216073] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6873.200477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.914107] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6881.173381] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.556911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6893.064727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 6895.848910] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6896.013333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6900.121373] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6900.232602] Lustre: lustre-OST0001: new disk, initializing [ 6900.237115] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6900.241601] Lustre: 221396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6904.253739] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6908.445932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6908.458577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6908.538789] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6910.456275] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.365462] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6913.496952] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6915.990679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6916.115732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6918.625781] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.630845] Lustre: Skipped 7 previous similar messages [ 6923.327869] Lustre: server umount lustre-OST0001 complete [ 6927.092454] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6927.239550] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6930.224868] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6930.382109] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6956.771210] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.747221] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6962.154527] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6962.161438] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6962.162321] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6962.173504] Lustre: Skipped 1 previous similar message [ 6965.252986] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6966.222465] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6971.834817] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 6971.843607] Lustre: Skipped 11 previous similar messages [ 6972.398930] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.408184] LustreError: Skipped 4 previous similar messages [ 6973.541896] Lustre: server umount lustre-OST0001 complete [ 6977.512929] LustreError: 219786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6977.535057] LustreError: 219786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 22 previous similar messages [ 6993.519920] Lustre: server umount lustre-OST0000 complete [ 7010.272118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7010.395916] Lustre: server umount lustre-MDT0000 complete [ 7012.572726] LustreError: 217946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192228 with bad export cookie 5045134150021862744 [ 7012.584908] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7012.585836] LustreError: 217946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7018.849341] Lustre: server umount lustre-MDT0001 complete [ 7032.352875] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:30:47 (1786192247) [ 7036.179783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.328049] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7036.345450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7038.931113] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7043.778491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7043.826585] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7043.841565] Lustre: 226645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7046.666225] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7051.122483] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7054.828865] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7058.645851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.737614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7058.842297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.848692] Lustre: Skipped 7 previous similar messages [ 7062.903930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7067.154051] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.473226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.539381] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7075.877371] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7079.606508] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.369685] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7082.659514] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7082.765529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7084.991259] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7085.116319] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7087.879878] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89cdc76f9800.ost_server_uuid 50 [ 7088.696862] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89cdc76f9800.ost_server_uuid in FULL state after 0 sec [ 7089.271714] Lustre: Permanently deactivating lustre-OST0001 [ 7089.280559] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7090.683857] Lustre: Permanently reactivating lustre-OST0001 [ 7098.828216] Lustre: server umount lustre-OST0001 complete [ 7115.232172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7115.333391] Lustre: server umount lustre-OST0000 complete [ 7123.421556] Lustre: server umount lustre-MDT0000 complete [ 7125.162126] LustreError: 227395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192340 with bad export cookie 5045134150021864025 [ 7125.165918] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7125.167180] LustreError: 227395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7125.327460] Lustre: server umount lustre-MDT0001 complete [ 7137.641406] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:32:32 (1786192352) [ 7141.541949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.724765] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7141.742870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7141.748071] Lustre: Skipped 5 previous similar messages [ 7144.463359] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7149.243065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.295753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7149.310677] Lustre: 233330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7151.594373] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7155.530409] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7158.188987] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7161.446403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7161.561524] Lustre: Permanently deactivating lustre-OST0000 [ 7161.569654] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7164.108431] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7164.881476] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7168.738710] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7172.156224] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.219669] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7178.819954] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7180.852637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7185.065242] Lustre: Permanently reactivating lustre-OST0000 [ 7185.073275] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7185.077179] Lustre: Skipped 2 previous similar messages [ 7187.146295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.153895] Lustre: Skipped 13 previous similar messages [ 7187.156900] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7187.159293] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7187.165374] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7187.184089] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7187.198266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7196.316323] Lustre: server umount lustre-OST0001 complete [ 7200.383547] Lustre: server umount lustre-OST0000 complete [ 7212.571248] Lustre: server umount lustre-MDT0000 complete [ 7214.641639] LustreError: 233323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192430 with bad export cookie 5045134150021865684 [ 7214.646361] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.881992] Lustre: server umount lustre-MDT0001 complete [ 7230.645568] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7232.148705] Key type lgssc unregistered [ 7232.315411] LNet: 238190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7232.317457] LNetError: 238190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7232.323345] LNet: Removed LNI 192.168.201.109@tcp [ 7232.726443] Key type .llcrypt unregistered [ 7232.727987] Key type ._llcrypt unregistered [ 7243.213343] Key type ._llcrypt registered [ 7243.214841] Key type .llcrypt registered [ 7243.261273] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7243.851035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.861779] alg: No test for adler32 (adler32-zlib) [ 7244.783450] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7244.940843] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7246.576152] Key type lgssc registered [ 7247.107117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7252.189580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7252.205449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.456287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7255.812982] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7260.168728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.227360] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7260.242599] Lustre: 239624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7260.249585] Lustre: Permanently deactivating lustre-MDT0001 [ 7260.260235] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7260.481255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7262.407794] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7263.226155] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7263.231218] LustreError: 239667:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7263.236517] LustreError: 239667:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7263.241655] LustreError: 239667:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 7266.048336] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7269.168116] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7272.955840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7273.149648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.545777] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7279.271630] LustreError: 242056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.276614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7279.282148] LustreError: 242056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7280.727719] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.525180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.620697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7285.687331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7285.687387] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7285.688358] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7288.191457] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7292.561762] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7295.718455] Lustre: Permanently reactivating lustre-MDT0001 [ 7295.739759] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7295.743211] Lustre: Skipped 2 previous similar messages [ 7300.530520] Lustre: lustre-MDT0001: Client 7a8146e0-5b0e-417f-a8f5-97483553402a (at 192.168.201.9@tcp) reconnecting [ 7304.339071] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7305.336482] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7308.074257] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7308.211851] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7310.496476] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7310.642947] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7312.136509] Lustre: Permanently deactivating lustre-MDT0001 [ 7312.146318] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7312.148311] Lustre: Skipped 2 previous similar messages [ 7318.556370] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7326.696209] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7326.701210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7326.702383] Lustre: Skipped 2 previous similar messages [ 7326.712389] Lustre: Skipped 3 previous similar messages [ 7331.173313] Lustre: server umount lustre-MDT0000 complete [ 7331.811833] LustreError: 239636:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.817562] LustreError: 239636:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7332.822266] LustreError: 239615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192548 with bad export cookie 11806445034293272129 [ 7332.825078] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.829552] LustreError: 239615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7332.977348] Lustre: server umount lustre-MDT0001 complete [ 7340.686596] Lustre: server umount lustre-OST0000 complete [ 7346.287234] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:36:01 (1786192561) [ 7351.442892] Lustre: server umount lustre-OST0001 complete [ 7354.526420] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7359.261617] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7376.973624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.134092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7377.136937] Lustre: Skipped 2 previous similar messages [ 7377.145614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7377.202375] Lustre: lustre-MDT0000: new disk, initializing [ 7377.248726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.255440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7379.218912] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7384.930542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.013954] Lustre: 247902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7385.058890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7385.062659] Lustre: Skipped 1 previous similar message [ 7385.131239] Lustre: lustre-MDT0001: new disk, initializing [ 7385.209359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7385.213842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7387.421963] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7392.554856] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7395.389476] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7398.770739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.937093] Lustre: lustre-OST0000: new disk, initializing [ 7398.939849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7398.943471] Lustre: 249869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7399.007409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.013971] Lustre: Skipped 1 previous similar message [ 7401.896691] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7404.558536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7404.570179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7404.639187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7406.858327] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7407.979699] LustreError: 247911:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7413.381656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.481744] Lustre: lustre-OST0001: new disk, initializing [ 7413.485335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7413.494922] Lustre: 250983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7416.533918] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7417.984216] LustreError: 247911:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7418.002780] LustreError: 250637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7421.573082] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7422.301247] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7423.019574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7423.023357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7423.057211] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7428.096352] LustreError: 250637:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7430.571690] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 7433.186390] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.191647] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.196037] Lustre: Skipped 1 previous similar message [ 7433.198773] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7433.206939] Lustre: Skipped 1 previous similar message [ 7435.185343] Lustre: server umount lustre-OST0001 complete [ 7435.686185] LustreError: 249853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.309145] LustreError: 249853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.317777] LustreError: 249853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7443.426985] LustreError: 250587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.435291] LustreError: 250587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7448.547670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.549950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7448.553424] Lustre: Skipped 2 previous similar messages [ 7453.666960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7453.672031] LustreError: 250587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.672649] Lustre: Skipped 2 previous similar messages [ 7453.680146] LustreError: 250587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7462.368241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7462.449849] Lustre: server umount lustre-OST0000 complete [ 7466.977600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7466.983325] LustreError: Skipped 1 previous similar message [ 7466.985065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7466.992140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7466.994984] Lustre: Skipped 2 previous similar messages [ 7470.499566] Lustre: server umount lustre-MDT0000 complete [ 7472.026336] LustreError: 247894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192687 with bad export cookie 11806445034293274936 [ 7472.026606] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7472.035059] LustreError: 247894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7478.245441] Lustre: server umount lustre-MDT0001 complete [ 7482.466642] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7483.802251] Key type lgssc unregistered [ 7483.952697] LNet: 253021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7483.957397] LNetError: 253021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7483.969528] LNet: Removed LNI 192.168.201.109@tcp [ 7484.417745] Key type .llcrypt unregistered [ 7484.419035] Key type ._llcrypt unregistered [ 7502.769490] Key type ._llcrypt registered [ 7502.770730] Key type .llcrypt registered [ 7502.824645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7503.422323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7503.430953] alg: No test for adler32 (adler32-zlib) [ 7504.332545] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7504.455793] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7506.056201] Key type lgssc registered [ 7506.678545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7510.554919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7510.569753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.699735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7511.721863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.816144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.652656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7517.632550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7517.698611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7517.720644] Lustre: 255205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7517.877407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.063196] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7523.419594] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7526.320361] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.718415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.838168] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7529.941977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7532.823151] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7532.899686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7536.390966] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7579.106669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7579.114130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7579.116743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7579.121023] Lustre: Skipped 1 previous similar message [ 7584.230170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7584.235215] Lustre: Skipped 1 previous similar message [ 7584.459172] Lustre: server umount lustre-OST0000 complete [ 7586.792742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7592.976888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7593.164376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.347567] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7600.138811] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7608.550550] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7608.555521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7608.555775] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7608.563436] Lustre: Skipped 1 previous similar message [ 7639.522910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.524183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7639.526880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7639.529696] Lustre: Skipped 2 previous similar messages [ 7639.537142] LustreError: Skipped 1 previous similar message [ 7640.696243] Lustre: server umount lustre-OST0000 complete [ 7644.642959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7644.647035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7644.650826] Lustre: Skipped 1 previous similar message [ 7644.657212] Lustre: Skipped 1 previous similar message [ 7648.948148] Lustre: server umount lustre-MDT0000 complete [ 7650.677117] LustreError: 255944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192866 with bad export cookie 17479283684607058009 [ 7650.678381] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7650.682037] LustreError: 255944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7650.814171] Lustre: server umount lustre-MDT0001 complete [ 7655.625113] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7657.009226] Key type lgssc unregistered [ 7657.166991] LNet: 260621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7657.170257] LNetError: 260621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7657.182374] LNet: Removed LNI 192.168.201.109@tcp [ 7657.605843] Key type .llcrypt unregistered [ 7657.607176] Key type ._llcrypt unregistered [ 7669.980716] Key type ._llcrypt registered [ 7669.982212] Key type .llcrypt registered [ 7670.037553] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7670.546123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7670.579620] alg: No test for adler32 (adler32-zlib) [ 7671.519431] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7671.661838] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7673.264243] Key type lgssc registered [ 7673.959969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7678.051571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7678.061546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.270916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.406476] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7686.070834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.289657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7688.550604] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7692.211718] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7694.808422] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7698.429928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7698.582384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.189587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7703.722446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7704.388253] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7719.393865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7719.396978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.403305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.443415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.444346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.444606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.446166] Lustre: Skipped 1 previous similar message [ 7725.397577] Lustre: server umount lustre-OST0000 complete [ 7727.585225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7727.588575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.593793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.682944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.683710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7733.279505] Lustre: server umount lustre-MDT0000 complete [ 7734.666452] LustreError: 261994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786192950 with bad export cookie 939314052478221436 [ 7734.669153] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7734.671929] LustreError: 261994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7734.788700] Lustre: server umount lustre-MDT0001 complete [ 7738.788857] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7739.909387] Key type lgssc unregistered [ 7740.048407] LNet: 266316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7740.050461] LNetError: 266316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7740.058330] LNet: Removed LNI 192.168.201.109@tcp [ 7740.402554] Key type .llcrypt unregistered [ 7740.404116] Key type ._llcrypt unregistered [ 7748.009863] Key type ._llcrypt registered [ 7748.011426] Key type .llcrypt registered [ 7748.060210] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7748.375030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7748.393466] alg: No test for adler32 (adler32-zlib) [ 7749.269950] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7749.368397] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7750.952239] Key type lgssc registered [ 7751.422578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.475479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7754.484969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7755.665124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.398950] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7760.925623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.135520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.684057] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7765.790780] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7768.174552] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7771.027790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7771.171989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.558557] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7776.362234] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.393539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7787.489387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7787.492256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.496872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.563472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7790.564261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.571888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7791.855215] Lustre: server umount lustre-OST0000 complete [ 7795.682907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.684277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7795.687522] Lustre: Skipped 1 previous similar message [ 7795.690140] Lustre: Skipped 1 previous similar message [ 7799.647212] Lustre: server umount lustre-MDT0000 complete [ 7800.801981] LustreError: 270261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.806702] LustreError: 270261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7801.074205] LustreError: 267497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193016 with bad export cookie 2025815681427082618 [ 7801.077405] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7801.078025] LustreError: 267497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7801.202923] Lustre: server umount lustre-MDT0001 complete [ 7805.238048] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7806.429777] Key type lgssc unregistered [ 7806.593508] LNet: 271193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7806.596182] LNetError: 271193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7806.603908] LNet: Removed LNI 192.168.201.109@tcp [ 7806.956257] Key type .llcrypt unregistered [ 7806.958087] Key type ._llcrypt unregistered [ 7817.389522] Key type ._llcrypt registered [ 7817.390789] Key type .llcrypt registered [ 7817.446429] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7817.883510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7817.932518] alg: No test for adler32 (adler32-zlib) [ 7818.816855] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7818.927435] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7820.528678] Key type lgssc registered [ 7821.073098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7824.724624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7824.739320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7825.952136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.844378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7831.935361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.136990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7834.094387] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7837.491278] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7840.030397] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.243197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.414270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.119488] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7849.796149] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.610813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7871.463623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7871.468282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.477688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.483407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.486420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.491639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.265495] Lustre: server umount lustre-OST0000 complete [ 7878.113342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7878.116397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.123854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.126606] Lustre: Skipped 1 previous similar message [ 7882.721529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.723405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.729307] Lustre: Skipped 1 previous similar message [ 7884.175377] Lustre: server umount lustre-MDT0000 complete [ 7885.870679] LustreError: 273321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193101 with bad export cookie 7014671884445522100 [ 7885.873155] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7885.879537] LustreError: 273321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7886.041106] Lustre: server umount lustre-MDT0001 complete [ 7890.425742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7891.570226] Key type lgssc unregistered [ 7891.710387] LNet: 276914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7891.713099] LNetError: 276914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7891.719381] LNet: Removed LNI 192.168.201.109@tcp [ 7892.048747] Key type .llcrypt unregistered [ 7892.050498] Key type ._llcrypt unregistered [ 7900.013926] Key type ._llcrypt registered [ 7900.016313] Key type .llcrypt registered [ 7900.060532] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7900.580136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7900.587686] alg: No test for adler32 (adler32-zlib) [ 7901.457094] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7901.561928] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7903.160102] Key type lgssc registered [ 7903.619217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7906.666443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7906.673296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7907.854755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.456639] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7912.651914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7912.776619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7914.126548] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7922.228679] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.552896] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.551852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.723275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.305756] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7932.840711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7933.418141] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7943.137792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7943.142075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.147216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.150127] Lustre: Skipped 1 previous similar message [ 7944.662241] Lustre: server umount lustre-OST0000 complete [ 7946.285912] Lustre: server umount lustre-MDT0000 complete [ 7947.802662] LustreError: 278093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193163 with bad export cookie 1113317014676847586 [ 7947.806819] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.808751] LustreError: 278093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.952622] Lustre: server umount lustre-MDT0001 complete [ 7952.208792] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7953.432025] Key type lgssc unregistered [ 7953.577575] LNet: 281824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7953.580689] LNetError: 281824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7953.589322] LNet: Removed LNI 192.168.201.109@tcp [ 7953.927870] Key type .llcrypt unregistered [ 7953.928991] Key type ._llcrypt unregistered [ 7958.401568] Key type ._llcrypt registered [ 7958.402878] Key type .llcrypt registered [ 7958.447778] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7964.687348] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7971.726331] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7972.095301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7972.148455] alg: No test for adler32 (adler32-zlib) [ 7973.039397] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 7973.148502] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7974.728191] Key type lgssc registered [ 7975.214704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7988.415668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7988.423467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7989.520548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7989.532037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7989.575107] Lustre: lustre-MDT0000: new disk, initializing [ 7989.608044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.614993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7991.166979] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7995.806867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.863877] Lustre: 285346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7995.910424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7995.913631] Lustre: Skipped 1 previous similar message [ 7995.954170] Lustre: lustre-MDT0001: new disk, initializing [ 7995.988345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7996.005910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7996.009083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7997.606441] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8002.090866] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8004.376054] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.368195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.507509] Lustre: lustre-OST0000: new disk, initializing [ 8007.511819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8007.516831] Lustre: 287316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8007.565204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.202189] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8010.751596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8010.755062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8010.795773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8014.646606] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8016.406498] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8016.488783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8018.161261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8018.251145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8020.963422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8020.963606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8020.967401] LustreError: Skipped 1 previous similar message [ 8020.968281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8020.973212] Lustre: Skipped 1 previous similar message [ 8026.082882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8026.085181] Lustre: Skipped 2 previous similar messages [ 8031.201752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8031.203805] Lustre: Skipped 1 previous similar message [ 8033.248191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8033.331231] Lustre: server umount lustre-OST0000 complete [ 8036.323590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.329048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.330437] Lustre: Skipped 1 previous similar message [ 8036.335420] Lustre: Skipped 1 previous similar message [ 8041.132690] Lustre: server umount lustre-MDT0000 complete [ 8041.441631] LustreError: 285354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.448409] LustreError: 285354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8042.676713] LustreError: 285341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193258 with bad export cookie 9824761288504369693 [ 8042.679586] LustreError: 285341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8042.679966] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8042.800656] Lustre: server umount lustre-MDT0001 complete [ 8046.022152] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:47:41 (1786193261) [ 8048.556416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8048.738964] LustreError: 289724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8048.765769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.201939] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8053.228837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.391421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8054.959656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8057.579430] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8059.739534] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.564616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.696456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.810600] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8067.621753] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8084.452502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8084.452720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8084.454942] LustreError: Skipped 1 previous similar message [ 8084.455615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8084.459961] Lustre: Skipped 1 previous similar message [ 8087.254303] Lustre: server umount lustre-OST0000 complete [ 8089.569548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8089.570018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8094.689187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8094.691625] Lustre: Skipped 4 previous similar messages [ 8094.810361] Lustre: server umount lustre-MDT0000 complete [ 8096.179128] LustreError: 290461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193311 with bad export cookie 9824761288504370575 [ 8096.180717] LustreError: 290464:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 8096.182211] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8096.184046] LustreError: 290461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8096.302990] Lustre: server umount lustre-MDT0001 complete [ 8100.078696] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8101.167943] Key type lgssc unregistered [ 8101.307500] LNet: 293281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8101.310306] LNetError: 293281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8101.323285] LNet: Removed LNI 192.168.201.109@tcp [ 8101.656657] Key type .llcrypt unregistered [ 8101.657984] Key type ._llcrypt unregistered [ 8110.923663] Key type ._llcrypt registered [ 8110.924991] Key type .llcrypt registered [ 8110.968684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8117.370731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8117.822742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8117.834462] alg: No test for adler32 (adler32-zlib) [ 8118.686920] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8118.778235] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8120.360211] Key type lgssc registered [ 8120.820094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.875883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8136.882398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.964229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8137.976454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8138.014935] Lustre: lustre-MDT0000: new disk, initializing [ 8138.037426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.044615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8139.453027] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8143.809729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.853617] Lustre: 297076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8143.878777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8143.881481] Lustre: Skipped 1 previous similar message [ 8143.918842] Lustre: lustre-MDT0001: new disk, initializing [ 8143.952124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.963836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8143.967141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8145.561840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8149.620911] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8151.584860] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8153.963420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.063980] Lustre: lustre-OST2710: new disk, initializing [ 8154.065783] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8154.068373] Lustre: 299041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8154.097796] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8156.098176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8156.102038] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8156.144831] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8156.160985] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8160.073670] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8162.508200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8162.573572] Lustre: lustre-OST03e8: new disk, initializing [ 8162.575401] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8162.578335] Lustre: 300108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8162.612305] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8163.016843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8163.020406] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8163.058661] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8164.761303] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8167.500568] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8170.348436] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8170.435089] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8171.996670] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8172.077695] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8173.608848] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8173.693415] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8175.211475] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8175.300403] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8178.660060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.662579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8178.665868] Lustre: Skipped 1 previous similar message [ 8182.562928] Lustre: server umount lustre-MDT0000 complete [ 8183.933260] LustreError: 300106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193399 with bad export cookie 9529801387099860416 [ 8183.936059] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8183.937331] LustreError: 300106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8184.074165] Lustre: server umount lustre-MDT0001 complete [ 8196.104928] Lustre: server umount lustre-OST2710 complete [ 8207.707283] Lustre: server umount lustre-OST03e8 complete [ 8214.160767] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8217.382027] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8229.373251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.461166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8229.470723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8229.509455] Lustre: lustre-MDT0000: new disk, initializing [ 8229.536864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.544537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8230.970622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8235.266799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.305398] Lustre: 305348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8235.396771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8236.788389] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8240.553266] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8242.531607] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8244.889464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8245.011464] Lustre: lustre-OST0000: new disk, initializing [ 8245.012930] Lustre: Skipped 1 previous similar message [ 8245.015498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8245.018139] Lustre: Skipped 2 previous similar messages [ 8245.020143] Lustre: 307316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8245.047891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8245.050484] Lustre: Skipped 1 previous similar message [ 8247.148506] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8250.870811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8250.873628] Lustre: Skipped 1 previous similar message [ 8250.876130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8250.905973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8251.023394] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8252.483709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8252.580703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8253.954776] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8254.027597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8255.970403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8255.970461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.972606] LustreError: Skipped 1 previous similar message [ 8255.972885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8255.972889] Lustre: Skipped 3 previous similar messages [ 8255.977567] Lustre: Skipped 3 previous similar messages [ 8260.816170] Lustre: server umount lustre-OST0000 complete [ 8266.210493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8266.210789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.220612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8266.222893] Lustre: Skipped 2 previous similar messages [ 8268.317766] Lustre: server umount lustre-MDT0000 complete [ 8269.537112] LustreError: 305341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193485 with bad export cookie 9529801387099862082 [ 8269.539415] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8269.541537] LustreError: 305341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8269.675804] Lustre: server umount lustre-MDT0001 complete [ 8272.731151] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:51:28 (1786193488) [ 8273.289871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8273.897153] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:51:29 (1786193489) [ 8275.058023] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8284.376449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.472488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8284.479851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8284.557402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.936587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8288.861794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.900637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8288.912361] Lustre: 311162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8290.448224] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8306.339218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8306.413443] Lustre: Denying initial registration attempt from nid 192.168.201.109@tcp, specified as failover [ 8306.417395] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8306.421561] LustreError: 312615:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8306.424597] LustreError: 312615:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8306.426568] LustreError: 312615:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8306.451904] Lustre: server umount lustre-OST0000 complete [ 8306.453664] LustreError: 312615:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8309.728843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8309.731104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.734414] Lustre: Skipped 1 previous similar message [ 8309.736051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8316.385639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8316.389522] Lustre: Skipped 3 previous similar messages [ 8323.040125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8323.097932] Lustre: server umount lustre-MDT0000 complete [ 8324.401248] LustreError: 311156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193540 with bad export cookie 9529801387099862950 [ 8324.404240] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8324.404910] LustreError: 311156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8324.518705] Lustre: server umount lustre-MDT0001 complete [ 8328.213738] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8329.313191] Key type lgssc unregistered [ 8329.449493] LNet: 313740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8329.451916] LNetError: 313740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8329.466365] LNet: Removed LNI 192.168.201.109@tcp [ 8329.785786] Key type .llcrypt unregistered [ 8329.786978] Key type ._llcrypt unregistered [ 8333.628417] Key type ._llcrypt registered [ 8333.629224] Key type .llcrypt registered [ 8333.664273] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8333.981196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8334.028061] alg: No test for adler32 (adler32-zlib) [ 8334.883847] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8334.975851] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8336.560130] Key type lgssc registered [ 8336.894890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8361.095120] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8364.271277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8364.278535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.358753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8365.366993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8365.421848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8366.733774] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8369.624603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.661271] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8369.672204] Lustre: 316479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8369.767939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8371.082966] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8373.427397] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8375.168106] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8390.306807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.368804] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8390.423087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.182139] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8394.405548] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8395.076085] Lustre: server umount lustre-OST0000 complete [ 8400.352728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8400.354909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.358085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8400.865397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.868675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8402.390439] Lustre: server umount lustre-MDT0000 complete [ 8403.512053] LustreError: 317221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193619 with bad export cookie 8257142276267649707 [ 8403.514654] LustreError: 317221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8403.514865] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8403.618561] Lustre: server umount lustre-MDT0001 complete [ 8407.249105] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8408.268673] Key type lgssc unregistered [ 8408.390554] LNet: 320047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8408.393146] LNetError: 320047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8408.400275] LNet: Removed LNI 192.168.201.109@tcp [ 8408.683201] Key type .llcrypt unregistered [ 8408.684245] Key type ._llcrypt unregistered [ 8417.314821] Key type ._llcrypt registered [ 8417.315756] Key type .llcrypt registered [ 8417.350854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8417.739689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8417.747913] alg: No test for adler32 (adler32-zlib) [ 8418.583837] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8418.664907] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8420.240140] Key type lgssc registered [ 8420.560456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8422.928661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8422.933562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.060267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8425.233476] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8427.896386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.009394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.214404] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8431.422133] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8433.146845] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8437.283326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.390746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8439.212041] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8441.457653] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8454.114511] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.115666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8454.118856] Lustre: Skipped 2 previous similar messages [ 8454.121415] Lustre: Skipped 2 previous similar messages [ 8457.128132] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 8459.234175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8459.236162] Lustre: Skipped 2 previous similar messages [ 8459.925923] Lustre: server umount lustre-MDT0000 complete [ 8461.055967] LustreError: 321425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193676 with bad export cookie 4652120066747548249 [ 8461.058689] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8461.058989] LustreError: 321425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8462.238729] LustreError: 323932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.244729] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 8462.245553] LustreError: 323932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8464.352670] LustreError: 321468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.353068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8466.400778] LustreError: 321439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.401064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8466.407589] Lustre: Skipped 1 previous similar message [ 8467.244066] Lustre: server umount lustre-MDT0001 complete [ 8469.294528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8471.952194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.856643] LustreError: 320751:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000eff0266 x1872959775989760/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8485.961780] LustreError: 323271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.984867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8486.004051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8487.269233] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8489.837570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.981971] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8491.090418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8493.161781] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8493.614094] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8495.193788] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8496.102930] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8502.777172] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 7 sec [ 8505.313108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8505.315565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.319319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8506.338268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8510.032725] Lustre: server umount lustre-OST0000 complete [ 8511.457429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.462010] Lustre: Skipped 2 previous similar messages [ 8517.400168] Lustre: server umount lustre-MDT0000 complete [ 8518.699118] LustreError: 321425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193734 with bad export cookie 4652120066747555403 [ 8518.702144] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8518.703887] LustreError: 321425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8518.831282] Lustre: server umount lustre-MDT0001 complete [ 8522.321202] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8523.303414] Key type lgssc unregistered [ 8523.425398] LNet: 327728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8523.427665] LNetError: 327728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8523.433370] LNet: Removed LNI 192.168.201.109@tcp [ 8523.755958] Key type .llcrypt unregistered [ 8523.757050] Key type ._llcrypt unregistered [ 8532.360359] Key type ._llcrypt registered [ 8532.361523] Key type .llcrypt registered [ 8532.397196] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8532.715834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8532.758065] alg: No test for adler32 (adler32-zlib) [ 8533.611129] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8533.694689] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8535.272195] Key type lgssc registered [ 8535.677838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8538.035447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8538.041379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.171857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.444366] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8543.007279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.131153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8544.282722] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8546.401972] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.055548] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8553.185940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.186914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8553.191319] Lustre: Skipped 1 previous similar message [ 8553.195737] Lustre: Skipped 1 previous similar message [ 8555.231889] Lustre: server umount lustre-MDT0000 complete [ 8556.447554] LustreError: 330736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193772 with bad export cookie 8456030156596068006 [ 8556.451261] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.585450] Lustre: server umount lustre-MDT0001 complete [ 8561.483250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8561.584369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8561.597180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8561.669791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.238789] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8566.214482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8566.246817] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8566.260754] Lustre: 331707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8566.358187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8567.809719] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8570.328559] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8571.987907] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8576.291392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8576.350768] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8576.353834] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8576.356993] LustreError: 333588:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 8576.359520] LustreError: 333588:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8576.361641] LustreError: 333588:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8576.385564] Lustre: server umount lustre-OST0000 complete [ 8576.387073] LustreError: 333588:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8580.963421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8581.038677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8581.100547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.993556] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8585.322130] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8587.794202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.829619] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8587.856498] Lustre: lustre-OST0001: new disk, initializing [ 8587.858052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8587.860212] Lustre: 335119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8588.946776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8588.950390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8588.974527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8588.998926] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8589.805322] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8592.255154] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8594.402437] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8594.405202] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8594.405402] LustreError: Skipped 1 previous similar message [ 8594.410716] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8594.413567] Lustre: Skipped 1 previous similar message [ 8595.326642] Lustre: server umount lustre-OST0001 complete [ 8599.524557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8599.526482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8599.527515] LustreError: 334054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.527524] LustreError: 334054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8599.528091] LustreError: Skipped 1 previous similar message [ 8599.528860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8599.531761] Lustre: Skipped 2 previous similar messages [ 8602.844694] Lustre: server umount lustre-OST0000 complete [ 8604.641117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.642274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8604.645502] Lustre: Skipped 1 previous similar message [ 8604.647873] Lustre: Skipped 2 previous similar messages [ 8609.761127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.763358] Lustre: Skipped 1 previous similar message [ 8610.260989] Lustre: server umount lustre-MDT0000 complete [ 8611.409949] LustreError: 331699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193827 with bad export cookie 8456030156596068545 [ 8611.412988] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8611.414112] LustreError: 331699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8611.521155] Lustre: server umount lustre-MDT0001 complete [ 8614.743703] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8615.685580] Key type lgssc unregistered [ 8615.822394] LNet: 337081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8615.824744] LNetError: 337081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8615.832320] LNet: Removed LNI 192.168.201.109@tcp [ 8616.143754] Key type .llcrypt unregistered [ 8616.145257] Key type ._llcrypt unregistered [ 8645.407870] Key type ._llcrypt registered [ 8645.408693] Key type .llcrypt registered [ 8645.439214] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8651.118304] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8651.497032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8651.535083] alg: No test for adler32 (adler32-zlib) [ 8652.384390] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8652.470870] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8654.056141] Key type lgssc registered [ 8654.379212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.167283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.172805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.246106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8666.255763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8666.289915] Lustre: lustre-MDT0000: new disk, initializing [ 8666.311091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.316658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8667.711840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8671.850674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8671.892465] Lustre: 341981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8671.914298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8671.916018] Lustre: Skipped 1 previous similar message [ 8671.952662] Lustre: lustre-MDT0001: new disk, initializing [ 8671.984919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8671.996825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8672.000624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8673.350042] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8677.059433] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8678.952409] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8681.096031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8681.194936] Lustre: lustre-OST0000: new disk, initializing [ 8681.196889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8681.199602] Lustre: 343949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8681.231725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.258759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8683.263575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8683.311585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8683.319727] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8687.057929] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.492519] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8688.568968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8689.893496] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8689.968599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8693.730214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.731299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.733728] LustreError: Skipped 1 previous similar message [ 8693.738436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8696.723201] Lustre: server umount lustre-OST0000 complete [ 8698.849820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.850426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.854182] Lustre: Skipped 2 previous similar messages [ 8698.855740] Lustre: Skipped 1 previous similar message [ 8703.968922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8703.970649] Lustre: Skipped 2 previous similar messages [ 8704.088922] Lustre: server umount lustre-MDT0000 complete [ 8705.317918] LustreError: 342776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786193920 with bad export cookie 1644429082765236768 [ 8705.321707] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.322105] LustreError: 342776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8705.432533] Lustre: server umount lustre-MDT0001 complete [ 8708.023354] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:58:43 (1786193923) [ 8710.272026] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:58:45 (1786193925) [ 8712.895825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.069278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8714.412967] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8717.164645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.298293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8718.699335] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8721.167617] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8722.992310] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8725.292616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.412562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8727.487871] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8729.992242] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8745.154683] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8745.693446] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8746.225624] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8746.785438] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8747.450762] Lustre: Failing over lustre-MDT0000 [ 8747.663276] Lustre: server umount lustre-MDT0000 complete [ 8748.002193] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.003387] LustreError: 346700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.007420] Lustre: Skipped 1 previous similar message [ 8748.012365] LustreError: 346700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8748.014271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8752.037457] LustreError: 346695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8752.043095] LustreError: 346695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8753.121909] LustreError: 346699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.128576] LustreError: 346699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8757.150843] LustreError: 349188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.550677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.593863] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8760.685115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8760.705945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8761.921901] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8762.271069] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8765.926031] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8765.933036] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 8765.949549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8767.685099] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8768.187175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8769.944572] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8770.543603] Lustre: Failing over lustre-MDT0000 [ 8770.774881] Lustre: server umount lustre-MDT0000 complete [ 8771.043229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8771.044321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.044715] LustreError: 348537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.044720] LustreError: 348537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8771.055462] Lustre: Skipped 3 previous similar messages [ 8781.281204] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.285954] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8783.662062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8783.705088] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8783.838256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8785.168271] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8787.871062] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8788.970746] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8788.973857] Lustre: Skipped 2 previous similar messages [ 8788.982951] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8789.003811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8790.826052] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8791.322498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8794.083072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.083326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.084983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.086479] Lustre: Skipped 2 previous similar messages [ 8799.202110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.205170] Lustre: Skipped 4 previous similar messages [ 8799.702212] Lustre: server umount lustre-MDT0000 complete [ 8800.948088] LustreError: 347433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194016 with bad export cookie 1644429082765239449 [ 8800.952158] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8800.952539] LustreError: 347433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8801.071501] Lustre: server umount lustre-MDT0001 complete [ 8811.545369] Lustre: server umount lustre-OST0000 complete [ 8816.340293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.475753] LustreError: 353050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8816.483143] LustreError: 353050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8816.495624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8816.498708] Lustre: Skipped 1 previous similar message [ 8817.703364] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8820.313913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.740296] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8823.855936] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8825.455616] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8827.510210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.453424] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8831.685343] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8841.955852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8846.305702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8846.308285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.311860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.422702] Lustre: server umount lustre-OST0000 complete [ 8856.723260] Lustre: server umount lustre-MDT0000 complete [ 8857.569540] LustreError: 353051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.574714] LustreError: 353051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8857.902195] LustreError: 353031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194073 with bad export cookie 1644429082765240170 [ 8857.904677] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8857.905463] LustreError: 353031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8858.014721] Lustre: server umount lustre-MDT0001 complete [ 8861.432353] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8862.396497] Key type lgssc unregistered [ 8862.512292] LNet: 356617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.514299] LNetError: 356617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.524297] LNet: Removed LNI 192.168.201.109@tcp [ 8862.806906] Key type .llcrypt unregistered [ 8862.807961] Key type ._llcrypt unregistered [ 8871.685071] Key type ._llcrypt registered [ 8871.685947] Key type .llcrypt registered [ 8871.720320] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8872.041421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8872.069547] alg: No test for adler32 (adler32-zlib) [ 8872.916293] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 8872.998282] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8874.576121] Key type lgssc registered [ 8874.876663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8877.318325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8877.323845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.446908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8879.605257] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8882.393142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.521316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8883.921265] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8886.355677] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8888.279396] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8890.565639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8890.693884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.678661] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8895.235022] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8904.040114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8913.377308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8913.380568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.386422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.401594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8914.403121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8914.406872] Lustre: Skipped 1 previous similar message [ 8915.604897] Lustre: server umount lustre-MDT0000 complete [ 8916.834529] LustreError: 358091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194132 with bad export cookie 804088057727777894 [ 8916.840553] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.968229] Lustre: server umount lustre-MDT0001 complete [ 8927.451401] Lustre: server umount lustre-OST0000 complete [ 8931.864803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.007322] LustreError: 361838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8932.014974] LustreError: 361838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8932.030632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8933.366172] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8936.174954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.297508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.800060] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8940.194950] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8941.990826] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8944.306908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.419616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.326192] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8948.726696] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8956.711078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8967.138454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8967.138982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8967.139738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8967.139742] Lustre: Skipped 2 previous similar messages [ 8967.148525] Lustre: Skipped 1 previous similar message [ 8972.251817] Lustre: server umount lustre-MDT0000 complete [ 8972.257963] LustreError: 361838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8972.262107] LustreError: 361838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8973.484139] LustreError: 361820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194189 with bad export cookie 804088057727778993 [ 8973.487610] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8973.489566] LustreError: 361820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8973.596523] Lustre: server umount lustre-MDT0001 complete [ 8985.048729] Lustre: server umount lustre-OST0000 complete [ 8990.073398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.249442] LustreError: 365764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8990.275564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8991.638714] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8994.539858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8996.058724] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8998.543303] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9000.495381] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9002.687765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9004.622389] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9007.018411] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9015.079476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9020.387969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.389723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9020.390123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9020.390131] Lustre: Skipped 2 previous similar messages [ 9020.391512] Lustre: Skipped 2 previous similar messages [ 9020.398275] LustreError: Skipped 1 previous similar message [ 9021.010650] Lustre: server umount lustre-OST0000 complete [ 9024.348850] Lustre: server umount lustre-MDT0000 complete [ 9025.505816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9025.505926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9025.508239] LustreError: 365764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.510960] Lustre: Skipped 1 previous similar message [ 9025.516196] LustreError: 365764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9025.563085] LustreError: 365745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194241 with bad export cookie 804088057727780001 [ 9025.565910] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9025.568693] LustreError: 365745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9025.674940] Lustre: server umount lustre-MDT0001 complete [ 9029.386128] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9030.391986] Key type lgssc unregistered [ 9030.510328] LNet: 369316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9030.512430] LNetError: 369316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9030.518301] LNet: Removed LNI 192.168.201.109@tcp [ 9030.824599] Key type .llcrypt unregistered [ 9030.826060] Key type ._llcrypt unregistered [ 9040.574375] Key type ._llcrypt registered [ 9040.575519] Key type .llcrypt registered [ 9040.612313] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9040.932316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9040.962545] alg: No test for adler32 (adler32-zlib) [ 9041.816158] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9041.903811] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9043.480160] Key type lgssc registered [ 9043.854589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9046.468487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9046.473591] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.476660] LustreError: 370735:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9046.483879] LustreError: 370735:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9046.487172] LustreError: 370735:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9046.489052] LustreError: 370735:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9046.491274] LustreError: 370735:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9048.962266] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9048.964677] LustreError: 371131:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9048.972911] LustreError: 371131:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9048.976885] LustreError: 371131:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9048.980846] LustreError: 371131:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9048.984413] LustreError: 371131:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9055.167551] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9056.146327] Key type lgssc unregistered [ 9056.274529] LNet: 372134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9056.276817] LNetError: 372134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9056.287259] LNet: Removed LNI 192.168.201.109@tcp [ 9056.588310] Key type .llcrypt unregistered [ 9056.589846] Key type ._llcrypt unregistered [ 9062.392735] Key type ._llcrypt registered [ 9062.393739] Key type .llcrypt registered [ 9062.430388] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9068.621365] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9069.086331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9069.094328] alg: No test for adler32 (adler32-zlib) [ 9069.944081] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9070.028475] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9071.608136] Key type lgssc registered [ 9071.958576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9083.143610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9083.152656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.238556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9084.248395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9084.288420] Lustre: lustre-MDT0000: new disk, initializing [ 9084.310418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.315806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9085.677178] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9089.733165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.772634] Lustre: 375261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9089.795279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9089.796878] Lustre: Skipped 1 previous similar message [ 9089.831897] Lustre: lustre-MDT0001: new disk, initializing [ 9089.865324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9089.879554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9089.883169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9091.174797] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9094.871709] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9096.704793] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9098.914837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.016601] Lustre: lustre-OST0000: new disk, initializing [ 9099.018400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9099.021162] Lustre: 377228:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9099.048553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9100.473921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9100.478398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9100.507658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9100.935356] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9104.558122] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9105.936101] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9106.016428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9107.341965] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9107.425168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9111.010832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9111.011462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.013873] LustreError: Skipped 1 previous similar message [ 9111.014521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9111.017532] Lustre: Skipped 1 previous similar message [ 9114.194188] Lustre: server umount lustre-OST0000 complete [ 9115.616900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9115.619187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9115.622889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9115.624424] Lustre: Skipped 1 previous similar message [ 9121.248976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9121.250681] Lustre: Skipped 3 previous similar messages [ 9121.561721] Lustre: server umount lustre-MDT0000 complete [ 9122.744566] LustreError: 377237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194338 with bad export cookie 1222217965962073015 [ 9122.747908] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9122.748331] LustreError: 377237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9122.871373] Lustre: server umount lustre-MDT0001 complete [ 9125.586035] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9126.126510] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:05:41 (1786194341) [ 9128.243702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9128.418442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9129.717464] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9132.417990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.536473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9133.875730] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9136.205396] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9138.000210] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9140.346917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9142.340742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9144.708306] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9146.920830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.982698] Lustre: lustre-OST0001: new disk, initializing [ 9146.984231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9146.986444] Lustre: 382516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9147.015545] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9147.017270] Lustre: Skipped 1 previous similar message [ 9148.891286] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9150.909845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9150.912686] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9150.941077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9152.580120] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9156.065774] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9156.066069] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.067845] LustreError: Skipped 1 previous similar message [ 9156.072094] Lustre: Skipped 1 previous similar message [ 9156.074801] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9156.077779] Lustre: Skipped 1 previous similar message [ 9160.580267] Lustre: server umount lustre-OST0001 complete [ 9164.263985] LustreError: 381885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9164.271554] LustreError: 381885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9169.380785] LustreError: 381885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9169.387737] LustreError: 381885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9173.474077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9173.477625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.483485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9173.484866] Lustre: Skipped 3 previous similar messages [ 9174.500958] LustreError: 381503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9174.505052] LustreError: 381503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9178.769440] Lustre: server umount lustre-OST0000 complete [ 9183.712857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.715696] LustreError: Skipped 1 previous similar message [ 9183.717471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.721916] Lustre: Skipped 1 previous similar message [ 9183.723947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9183.726525] Lustre: Skipped 2 previous similar messages [ 9186.203938] Lustre: server umount lustre-MDT0000 complete [ 9187.412230] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194403 with bad export cookie 1222217965962073890 [ 9187.415412] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9187.415564] LustreError: 379665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9193.577259] Lustre: server umount lustre-MDT0001 complete [ 9197.423453] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9198.543254] Key type lgssc unregistered [ 9198.688391] LNet: 384505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9198.690881] LNetError: 384505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9198.699300] LNet: Removed LNI 192.168.201.109@tcp [ 9199.022165] Key type .llcrypt unregistered [ 9199.023479] Key type ._llcrypt unregistered [ 9216.821435] Key type ._llcrypt registered [ 9216.822503] Key type .llcrypt registered [ 9216.859384] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9217.250807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9217.278930] alg: No test for adler32 (adler32-zlib) [ 9218.148830] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9218.244444] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9219.824143] Key type lgssc registered [ 9220.228047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9223.022747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9223.028890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.114761] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9224.130109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9224.200340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9225.703920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9228.785160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.820043] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9228.830800] Lustre: 387276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9228.934962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9230.382679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9232.958465] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9234.887148] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9239.520982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9239.523440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.527189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.033165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9240.033716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.691083] Lustre: server umount lustre-MDT0000 complete [ 9242.912395] LustreError: 387268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194458 with bad export cookie 3017055680391789040 [ 9242.915403] LustreError: 387268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9242.915463] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9243.030701] Lustre: server umount lustre-MDT0001 complete [ 9246.238647] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:07:41 (1786194461) [ 9248.534846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.720651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9250.158828] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9253.140544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.269294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9254.643539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9257.085441] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9258.925343] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9261.119734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.194439] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9261.255640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9263.195603] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9265.556053] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9269.411400] Lustre: Permanently deactivating lustre-OST0000 [ 9269.416189] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9271.665169] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9272.625460] LustreError: 392671:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9272.628920] LustreError: 392671:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9272.631121] LustreError: 392671:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9273.247353] LustreError: 392719:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9273.250746] LustreError: 392719:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9280.020967] Lustre: server umount lustre-OST0000 complete [ 9280.959835] LustreError: 392968:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9280.962353] LustreError: 392968:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9284.064771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9284.067126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9284.071313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.073167] Lustre: Skipped 1 previous similar message [ 9289.697425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9289.699459] Lustre: Skipped 2 previous similar messages [ 9291.768818] Lustre: server umount lustre-MDT0000 complete [ 9293.019309] LustreError: 389811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194508 with bad export cookie 3017055680391789607 [ 9293.022594] LustreError: 389811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9293.023168] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9302.545786] Lustre: server umount lustre-MDT0001 complete [ 9305.610325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9306.962346] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9307.901736] LustreError: 394126:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9309.968066] Lustre: 394464:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.109@tcp [ 9310.657948] Lustre: server umount MGS complete [ 9314.489880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9314.638292] LustreError: 395187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.642461] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9314.642722] LustreError: 395187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9314.644420] Lustre: Skipped 1 previous similar message [ 9314.665294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9316.135123] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9318.874645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.970109] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9320.296802] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9322.664247] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9324.434741] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9325.087944] Lustre: Permanently reactivating lustre-OST0000 [ 9325.093109] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9325.094919] Lustre: Skipped 2 previous similar messages [ 9353.696102] LustreError: 397089:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9355.928333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9356.034977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9356.037105] Lustre: Skipped 1 previous similar message [ 9357.948854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9360.404157] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9373.665066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9373.667433] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9373.671240] Lustre: Skipped 1 previous similar message [ 9373.673431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9378.507686] Lustre: server umount lustre-OST0000 complete [ 9379.809276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.815137] Lustre: Skipped 1 previous similar message [ 9384.928974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9384.931608] Lustre: Skipped 5 previous similar messages [ 9394.144162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9394.206255] Lustre: server umount lustre-MDT0000 complete [ 9395.169025] LustreError: 395187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.175413] LustreError: 395187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9395.446528] LustreError: 395212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194611 with bad export cookie 3017055680391790629 [ 9395.449952] LustreError: 395212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9395.450212] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.576192] Lustre: server umount lustre-MDT0001 complete [ 9399.362117] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9400.431849] Key type lgssc unregistered [ 9400.574533] LNet: 399957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9400.577307] LNetError: 399957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9400.588564] LNet: Removed LNI 192.168.201.109@tcp [ 9400.944283] Key type .llcrypt unregistered [ 9400.946042] Key type ._llcrypt unregistered [ 9406.756509] Key type ._llcrypt registered [ 9406.757767] Key type .llcrypt registered [ 9406.796442] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9412.812782] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9413.192620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9413.247140] alg: No test for adler32 (adler32-zlib) [ 9414.095787] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9414.183040] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9415.768271] Key type lgssc registered [ 9416.111045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9427.136088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9427.141391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.224892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9428.234881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9428.274214] Lustre: lustre-MDT0000: new disk, initializing [ 9428.296501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9428.302188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9429.667852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9433.739223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9433.782699] Lustre: 403086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9433.804225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9433.806030] Lustre: Skipped 1 previous similar message [ 9433.841818] Lustre: lustre-MDT0001: new disk, initializing [ 9433.872337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9433.884032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9433.888239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9435.150984] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9438.775449] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9440.611195] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9442.741077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9442.845434] Lustre: lustre-OST0000: new disk, initializing [ 9442.846873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9442.849235] Lustre: 405053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9442.883527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9444.909387] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9448.606850] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9449.977406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9449.981381] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9450.024755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9450.037421] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9450.110658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9451.414913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9451.489495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9455.075958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9455.075995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.080091] LustreError: Skipped 1 previous similar message [ 9455.083972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9455.087565] Lustre: Skipped 1 previous similar message [ 9458.258466] Lustre: server umount lustre-OST0000 complete [ 9459.680843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9459.683236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9459.688877] Lustre: Skipped 1 previous similar message [ 9459.690431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.313250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.314814] Lustre: Skipped 3 previous similar messages [ 9465.632660] Lustre: server umount lustre-MDT0000 complete [ 9466.896208] LustreError: 403079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194682 with bad export cookie 14660402713657966433 [ 9466.900238] LustreError: 403079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9466.900309] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9467.035508] Lustre: server umount lustre-MDT0001 complete [ 9469.794081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9470.371530] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:11:25 (1786194685) [ 9472.566898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9472.734410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9474.130583] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9476.935049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.065724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9478.474780] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9480.874571] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9482.642086] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9484.920571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9487.037688] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9489.483898] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9490.092922] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9501.155778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9502.692467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.694604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.697820] Lustre: Skipped 1 previous similar message [ 9502.699942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.701499] Lustre: Skipped 1 previous similar message [ 9507.811885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9507.814681] Lustre: Skipped 1 previous similar message [ 9508.622216] Lustre: server umount lustre-OST0000 complete [ 9512.930215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.930373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.933876] Lustre: Skipped 2 previous similar messages [ 9512.935749] LustreError: Skipped 1 previous similar message [ 9518.049030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9518.050601] Lustre: Skipped 3 previous similar messages [ 9524.192105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9524.248361] Lustre: server umount lustre-MDT0000 complete [ 9525.439968] LustreError: 407489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194741 with bad export cookie 14660402713657967301 [ 9525.442829] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9525.443144] LustreError: 407489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9525.558807] Lustre: server umount lustre-MDT0001 complete [ 9528.987958] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9529.990642] Key type lgssc unregistered [ 9530.111318] LNet: 411259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9530.113441] LNetError: 411259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9530.124307] LNet: Removed LNI 192.168.201.109@tcp [ 9530.430852] Key type .llcrypt unregistered [ 9530.431949] Key type ._llcrypt unregistered [ 9537.749808] Key type ._llcrypt registered [ 9537.750707] Key type .llcrypt registered [ 9537.790538] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9537.900558] Key type .llcrypt unregistered [ 9537.901483] Key type ._llcrypt unregistered [ 9544.602402] Key type ._llcrypt registered [ 9544.603375] Key type .llcrypt registered [ 9544.639890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9545.047503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9545.054748] alg: No test for adler32 (adler32-zlib) [ 9545.904851] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9546.004923] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9547.584180] Key type lgssc registered [ 9547.947037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9550.484496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9550.491219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9551.630920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9552.989984] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9555.739848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9555.858974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.728902] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9560.070388] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9561.057447] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9561.062284] LustreError: 413449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9562.081320] LustreError: 413448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.086574] LustreError: 413448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9562.086957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9562.269276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.386473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9563.670684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9572.321045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9572.324340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.330737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.836078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9572.836833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.843688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9576.972550] Lustre: server umount lustre-OST0000 complete [ 9582.050910] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 9582.054270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.059902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9584.351177] Lustre: server umount lustre-MDT0000 complete [ 9585.624381] LustreError: 415189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194801 with bad export cookie 6273314700228836102 [ 9585.629077] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9585.629084] LustreError: 415189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9585.761245] Lustre: server umount lustre-MDT0001 complete [ 9589.293071] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9590.336636] Key type lgssc unregistered [ 9590.467572] LNet: 416597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9590.470172] LNetError: 416597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9590.477390] LNet: Removed LNI 192.168.201.109@tcp [ 9590.818675] Key type .llcrypt unregistered [ 9590.820361] Key type ._llcrypt unregistered [ 9599.551284] Key type ._llcrypt registered [ 9599.552449] Key type .llcrypt registered [ 9599.591452] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9599.967383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9600.017738] alg: No test for adler32 (adler32-zlib) [ 9600.867743] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9600.958777] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9602.544111] Key type lgssc registered [ 9602.900063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9605.413298] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9605.419746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9620.832360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9625.952254] LustreError: 417964:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 9652.576170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.491814] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9656.973027] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9659.307538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.479241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9659.502360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9660.867678] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9663.823348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.970308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9665.319421] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9667.741574] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9669.487408] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9672.163133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9672.168449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9672.170293] Lustre: Skipped 1 previous similar message [ 9677.281938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9677.844163] Lustre: server umount lustre-OST0000 complete [ 9679.328872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9679.331290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9679.336422] Lustre: Skipped 1 previous similar message [ 9679.338484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9679.340541] Lustre: Skipped 1 previous similar message [ 9682.401120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9682.401781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.274155] Lustre: server umount lustre-MDT0000 complete [ 9686.493308] LustreError: 419726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194902 with bad export cookie 8932844742072236998 [ 9686.496786] LustreError: 419726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9686.497324] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9686.613888] Lustre: server umount lustre-MDT0001 complete [ 9690.148454] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9691.181047] Key type lgssc unregistered [ 9691.300409] LNet: 421534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9691.302610] LNetError: 421534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9691.313289] LNet: Removed LNI 192.168.201.109@tcp [ 9691.611093] Key type .llcrypt unregistered [ 9691.612510] Key type ._llcrypt unregistered [ 9700.522819] Key type ._llcrypt registered [ 9700.524252] Key type .llcrypt registered [ 9700.563890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9700.985595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9700.996944] alg: No test for adler32 (adler32-zlib) [ 9701.857745] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9701.949869] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9703.536105] Key type lgssc registered [ 9703.888083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9706.520176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9706.527288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.701802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.075562] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9711.972523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.093342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9713.487694] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9715.879775] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9717.672708] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9720.009198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9720.118698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.989175] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9724.415096] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9732.391959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9737.701627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9737.702465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9737.702681] Lustre: Skipped 1 previous similar message [ 9737.704132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9737.716360] Lustre: Skipped 2 previous similar messages [ 9740.199704] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 9742.169092] Lustre: server umount lustre-MDT0000 complete [ 9742.819678] LustreError: 422927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.823825] LustreError: 422927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9742.952481] LustreError: 424768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.9@tcp arrived at 1786194958 with bad export cookie 8559595583598409946 [ 9742.957900] LustreError: 424768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9743.579238] LustreError: 422908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786194959 with bad export cookie 8559595583598409295 [ 9743.583608] LustreError: 422908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9743.583681] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9747.937510] LustreError: 423655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9747.937895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.937941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9747.944145] LustreError: 423655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9747.947517] Lustre: Skipped 1 previous similar message [ 9747.954504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9748.961090] LustreError: 422926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.967557] LustreError: 422926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9749.712274] Lustre: server umount lustre-OST0000 complete [ 9752.007626] Lustre: server umount lustre-MDT0001 complete [ 9755.746332] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9756.829538] Key type lgssc unregistered [ 9756.955520] LNet: 426630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9756.957959] LNetError: 426630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9756.965425] LNet: Removed LNI 192.168.201.109@tcp [ 9757.304099] Key type .llcrypt unregistered [ 9757.305208] Key type ._llcrypt unregistered [ 9766.389524] Key type ._llcrypt registered [ 9766.390674] Key type .llcrypt registered [ 9766.429355] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9766.791823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9766.805805] alg: No test for adler32 (adler32-zlib) [ 9767.682805] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9767.808194] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9769.408194] Key type lgssc registered [ 9769.806139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9772.455957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9772.462813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.644548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9774.966263] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9777.813807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9777.956568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9779.388033] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9781.749212] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9783.622586] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9785.946447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.081637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.017391] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9790.799235] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9798.375663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9803.747689] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.747975] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9803.751728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9803.755335] Lustre: Skipped 1 previous similar message [ 9806.247096] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 9808.144877] Lustre: server umount lustre-MDT0001 complete [ 9808.867300] LustreError: 428016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.871756] LustreError: 428016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9813.986952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9813.987301] LustreError: 428021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.987448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.987458] Lustre: Skipped 1 previous similar message [ 9813.988233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9815.888750] Lustre: server umount lustre-OST0000 complete [ 9817.266985] Lustre: server umount lustre-MDT0000 complete [ 9821.913489] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9823.019303] Key type lgssc unregistered [ 9823.153516] LNet: 431731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9823.156309] LNetError: 431731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9823.165404] LNet: Removed LNI 192.168.201.109@tcp [ 9823.485829] Key type .llcrypt unregistered [ 9823.486975] Key type ._llcrypt unregistered [ 9832.114592] Key type ._llcrypt registered [ 9832.115635] Key type .llcrypt registered [ 9832.153619] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9838.585241] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9839.016570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9839.063407] alg: No test for adler32 (adler32-zlib) [ 9839.927189] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9840.027196] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9841.616197] Key type lgssc registered [ 9841.965899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9855.389370] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9860.712447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9860.717902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.808119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9861.820112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9861.867961] Lustre: lustre-MDT0000: new disk, initializing [ 9861.892151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.897895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9863.298975] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9869.194189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9869.313080] Lustre: lustre-OST0000: new disk, initializing [ 9869.314614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9869.316050] Lustre: Skipped 1 previous similar message [ 9869.317760] Lustre: 436735:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9869.342990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9871.251654] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9871.812898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9871.815611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9871.824212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9874.881567] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9880.326643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.375191] Lustre: 435718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9880.404783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9880.444927] Lustre: lustre-MDT0001: new disk, initializing [ 9880.494145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.508971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9880.512185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9881.899825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9885.546013] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9885.617703] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9887.066926] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9890.227155] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 9893.285747] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 9894.369783] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9894.371686] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.373170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9894.378333] Lustre: Skipped 1 previous similar message [ 9898.399500] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 9898.401802] Lustre: Skipped 1 previous similar message [ 9903.519921] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 9903.523947] Lustre: Skipped 2 previous similar messages [ 9906.144113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9906.212989] Lustre: server umount lustre-MDT0001 complete [ 9914.852332] LustreError: 436743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.859812] LustreError: 436743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9919.970602] LustreError: 435728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9919.970710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.979858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9919.981528] Lustre: Skipped 2 previous similar messages [ 9924.632742] Lustre: server umount lustre-OST0000 complete [ 9925.091250] LustreError: 435727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9932.225523] Lustre: server umount lustre-MDT0000 complete [ 9936.726942] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9937.764491] Key type lgssc unregistered [ 9937.886411] LNet: 440504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9937.889587] LNetError: 440504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9937.898310] LNet: Removed LNI 192.168.201.109@tcp [ 9938.256680] Key type .llcrypt unregistered [ 9938.257889] Key type ._llcrypt unregistered [ 9951.402523] Key type ._llcrypt registered [ 9951.403534] Key type .llcrypt registered [ 9951.440859] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9951.556198] Key type .llcrypt unregistered [ 9951.557269] Key type ._llcrypt unregistered [ 9958.001461] Key type ._llcrypt registered [ 9958.002469] Key type .llcrypt registered [ 9958.041915] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9958.383405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9958.448405] alg: No test for adler32 (adler32-zlib) [ 9959.300915] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 9959.399776] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9960.976126] Key type lgssc registered [ 9961.354628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9970.079862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9970.085719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9971.177810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9971.188668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9971.227713] Lustre: lustre-MDT0000: new disk, initializing [ 9971.249221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9971.256811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9972.627139] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9976.776141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.824721] Lustre: 443856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9976.850735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9976.852840] Lustre: Skipped 1 previous similar message [ 9976.894829] Lustre: lustre-MDT0001: new disk, initializing [ 9976.930600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.943461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9976.947038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9978.350745] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9982.182359] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9984.076611] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9986.366961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.502316] Lustre: lustre-OST0000: new disk, initializing [ 9986.503956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9986.506328] Lustre: 445821:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9986.537513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9988.431839] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9992.176829] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9993.720794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9993.723932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9993.755664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10002.401189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10002.405090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.411766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.937801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.939417] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10003.945636] Lustre: Skipped 1 previous similar message [10006.048522] Lustre: server umount lustre-MDT0000 complete [10007.312503] LustreError: 443894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195222 with bad export cookie 16490662220091464681 [10007.316202] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10007.316709] LustreError: 443894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10007.432201] Lustre: server umount lustre-MDT0001 complete [10018.975786] Lustre: server umount lustre-OST0000 complete [10022.487525] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:20:37 (1786195237) [10025.094599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.288214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10026.655577] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10029.575439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.694246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10031.155709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10033.739783] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10035.704888] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10051.043254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10051.123685] Lustre: Found index 0 for lustre-OST0000, updating log [10051.190357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10053.135920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10055.595364] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10070.633630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10076.132977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10076.134706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10076.137238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10076.139917] Lustre: Skipped 1 previous similar message [10076.141863] Lustre: Skipped 2 previous similar messages [10078.802379] Lustre: server umount lustre-OST0000 complete [10091.376737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.377573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10091.380596] Lustre: Skipped 1 previous similar message [10091.382758] Lustre: Skipped 3 previous similar messages [10096.609513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10096.611365] Lustre: Skipped 2 previous similar messages [10097.500524] Lustre: server umount lustre-MDT0000 complete [10098.757477] LustreError: 447828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195314 with bad export cookie 16490662220091466046 [10098.761073] LustreError: 447828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10098.761521] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10098.881600] Lustre: server umount lustre-MDT0001 complete [10101.841816] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:21:57 (1786195317) [10120.160279] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [10681.085866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.246426] LustreError: 454268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.251863] LustreError: 454268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10681.274360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10682.584831] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10685.310881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.433147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10686.731402] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10688.936026] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10690.593923] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10705.762898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10705.862335] Lustre: Found index 0 for lustre-OST0000, updating log [10705.949437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10707.594646] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10708.007199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10709.690664] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10711.011478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10711.015290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10716.369771] Lustre: server umount lustre-OST0000 complete [10718.625637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.628776] Lustre: Skipped 1 previous similar message [10723.733650] Lustre: server umount lustre-MDT0000 complete [10723.745215] LustreError: 454269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10723.749195] LustreError: 454269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10724.769332] LustreError: 454268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.773324] LustreError: 454268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10724.926321] LustreError: 456166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786195940 with bad export cookie 16490662220091467117 [10724.929236] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10724.929352] LustreError: 456166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10725.054066] Lustre: server umount lustre-MDT0001 complete [10735.584258] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [10742.055657] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [10745.076452] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10756.550728] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:32:51 (1786195971) [10762.028130] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10763.071176] Key type lgssc unregistered [10768.352495] LNet: 460921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10768.354702] LNetError: 460921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10835.942319] LNet: Removed LNI 192.168.201.109@tcp [10836.278043] Key type .llcrypt unregistered [10836.279050] Key type ._llcrypt unregistered [10842.438296] Key type ._llcrypt registered [10842.439344] Key type .llcrypt registered [10842.471625] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10842.770141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10842.778835] alg: No test for adler32 (adler32-zlib) [10843.623952] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [10843.709529] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10845.288136] Key type lgssc registered [10845.663487] Lustre: Echo OBD driver; http://www.lustre.org/ [10847.088193] LNet: Added LNI 192.168.201.109@tcp100 [8/256/0/180] [10847.371191] LNet: Added LNI 192.168.201.109@tcp101 [8/256/0/180] [10847.661508] LNet: Added LNI 192.168.201.109@tcp102 [8/256/0/180] [10847.961533] LNet: Added LNI 192.168.201.109@tcp103 [8/256/0/180] [10848.252370] LNet: Added LNI 192.168.201.109@tcp104 [8/256/0/180] [10848.553199] LNet: Added LNI 192.168.201.109@tcp105 [8/256/0/180] [10848.862879] LNet: Added LNI 192.168.201.109@tcp106 [8/256/0/180] [10849.164212] LNet: Added LNI 192.168.201.109@tcp107 [8/256/0/180] [10849.462809] LNet: Added LNI 192.168.201.109@tcp108 [8/256/0/180] [10849.746435] LNet: Added LNI 192.168.201.109@tcp109 [8/256/0/180] [10850.034787] LNet: Added LNI 192.168.201.109@tcp110 [8/256/0/180] [10850.329826] LNet: Added LNI 192.168.201.109@tcp111 [8/256/0/180] [10850.630657] LNet: Added LNI 192.168.201.109@tcp112 [8/256/0/180] [10850.911866] LNet: Added LNI 192.168.201.109@tcp113 [8/256/0/180] [10851.198164] LNet: Added LNI 192.168.201.109@tcp114 [8/256/0/180] [10851.489657] LNet: Added LNI 192.168.201.109@tcp115 [8/256/0/180] [10851.779407] LNet: Added LNI 192.168.201.109@tcp116 [8/256/0/180] [10852.067645] LNet: Added LNI 192.168.201.109@tcp117 [8/256/0/180] [10852.347136] LNet: Added LNI 192.168.201.109@tcp118 [8/256/0/180] [10852.631523] LNet: Added LNI 192.168.201.109@tcp119 [8/256/0/180] [10852.938941] LNet: Added LNI 192.168.201.109@tcp120 [8/256/0/180] [10853.234780] LNet: Added LNI 192.168.201.109@tcp121 [8/256/0/180] [10853.516730] LNet: Added LNI 192.168.201.109@tcp122 [8/256/0/180] [10853.805801] LNet: Added LNI 192.168.201.109@tcp123 [8/256/0/180] [10854.108123] LNet: Added LNI 192.168.201.109@tcp124 [8/256/0/180] [10854.394314] LNet: Added LNI 192.168.201.109@tcp125 [8/256/0/180] [10854.685961] LNet: Added LNI 192.168.201.109@tcp126 [8/256/0/180] [10854.980257] LNet: Added LNI 192.168.201.109@tcp127 [8/256/0/180] [10855.260291] LNet: Added LNI 192.168.201.109@tcp128 [8/256/0/180] [10855.547850] LNet: Added LNI 192.168.201.109@tcp129 [8/256/0/180] [10855.829840] LNet: Added LNI 192.168.201.109@tcp130 [8/256/0/180] [10856.110242] LNet: Added LNI 192.168.201.109@tcp131 [8/256/0/180] [10856.399757] LNet: Added LNI 192.168.201.109@tcp132 [8/256/0/180] [10856.686751] LNet: Added LNI 192.168.201.109@tcp133 [8/256/0/180] [10856.970740] LNet: Added LNI 192.168.201.109@tcp134 [8/256/0/180] [10857.253722] LNet: Added LNI 192.168.201.109@tcp135 [8/256/0/180] [10859.377572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10859.382373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.419767] Lustre: 463920:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [10860.447265] LNetError: 461381:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [10860.452656] LNetError: Protocol error connecting to 192.168.201.109@tcp115 at host 192.168.201.109:988: is it running a compatible version of Lustre? [10860.457071] LNetError: Skipped 3 previous similar messages [10860.497994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10860.507846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10860.545984] Lustre: lustre-MDT0000: new disk, initializing [10860.568395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10860.577867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10861.953379] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10865.905986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10865.952775] Lustre: 463938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10865.981345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10865.983014] Lustre: Skipped 1 previous similar message [10866.016954] Lustre: lustre-MDT0001: new disk, initializing [10866.028450] Lustre: 464739:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [10866.030720] Lustre: 464739:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10866.068810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10866.082447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10866.085749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10867.302737] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10870.064926] Lustre: 465268:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [10870.068850] Lustre: 465268:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10870.857582] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10872.564846] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10874.713766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10874.839890] Lustre: lustre-OST0000: new disk, initializing [10874.841398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10874.843569] Lustre: 465909:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10874.872768] Lustre: 465914:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [10874.875122] Lustre: 465914:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10874.889996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10876.801820] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10876.931336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10876.934082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10876.997661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10880.415834] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10887.137769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10887.138572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.140362] LustreError: Skipped 1 previous similar message [10887.146354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10887.148973] Lustre: Skipped 1 previous similar message [10888.784445] Lustre: server umount lustre-OST0000 complete [10891.744771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10891.746844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10891.749970] Lustre: Skipped 1 previous similar message [10891.751645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10892.896864] LNetError: 463942:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [10892.899242] LNetError: 463942:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [10892.901112] LNetError: Protocol error connecting to 192.168.201.109@tcp116 at host 192.168.201.109:988: is it running a compatible version of Lustre? [10894.945353] LNetError: 463941:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [10894.950222] LNetError: Protocol error connecting to 192.168.201.109@tcp119 at host 192.168.201.109:988: is it running a compatible version of Lustre? [10896.098563] Lustre: server umount lustre-MDT0000 complete [10897.278233] LustreError: 463930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196112 with bad export cookie 1563920182185893324 [10897.281311] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10897.281634] LustreError: 463930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10897.397288] Lustre: server umount lustre-MDT0001 complete [10898.018487] LNetError: 463948:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [10898.021034] LNetError: Protocol error connecting to 192.168.201.109@tcp120 at host 192.168.201.109:988: is it running a compatible version of Lustre? [10901.032609] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10902.047652] Key type lgssc unregistered [10902.205304] LNet: 467825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10902.209474] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10902.213245] LNet: Removed LNI 192.168.201.109@tcp [10903.264146] LNet: Removed LNI 192.168.201.109@tcp100 [10903.265364] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10903.267204] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10903.269115] LNet: Removed LNI 192.168.201.109@tcp101 [10904.288142] LNet: Removed LNI 192.168.201.109@tcp102 [10904.289446] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10904.291381] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10904.293665] LNet: Removed LNI 192.168.201.109@tcp103 [10905.312119] LNet: Removed LNI 192.168.201.109@tcp104 [10906.336145] LNet: Removed LNI 192.168.201.109@tcp105 [10906.338190] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10906.341399] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10906.345084] LNet: Removed LNI 192.168.201.109@tcp106 [10907.360154] LNet: Removed LNI 192.168.201.109@tcp107 [10908.384127] LNet: Removed LNI 192.168.201.109@tcp108 [10909.408143] LNet: Removed LNI 192.168.201.109@tcp109 [10910.432129] LNet: Removed LNI 192.168.201.109@tcp110 [10910.434442] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10910.436571] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10910.439176] LNet: Removed LNI 192.168.201.109@tcp111 [10911.456147] LNet: Removed LNI 192.168.201.109@tcp112 [10912.480105] LNet: Removed LNI 192.168.201.109@tcp113 [10913.504139] LNet: Removed LNI 192.168.201.109@tcp114 [10914.528116] LNet: Removed LNI 192.168.201.109@tcp115 [10915.552116] LNet: Removed LNI 192.168.201.109@tcp116 [10916.576092] LNet: Removed LNI 192.168.201.109@tcp117 [10917.600123] LNet: Removed LNI 192.168.201.109@tcp118 [10918.624111] LNet: Removed LNI 192.168.201.109@tcp119 [10918.625341] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10918.627266] LNetError: 467825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10918.630220] LNet: Removed LNI 192.168.201.109@tcp120 [10918.631364] LNet: Removed LNI 192.168.201.109@tcp121 [10918.632405] LNet: Removed LNI 192.168.201.109@tcp122 [10918.633457] LNet: Removed LNI 192.168.201.109@tcp123 [10918.634425] LNet: Removed LNI 192.168.201.109@tcp124 [10918.635442] LNet: Removed LNI 192.168.201.109@tcp125 [10918.636449] LNet: Removed LNI 192.168.201.109@tcp126 [10918.637453] LNet: Removed LNI 192.168.201.109@tcp127 [10918.638508] LNet: Removed LNI 192.168.201.109@tcp128 [10918.639875] LNet: Removed LNI 192.168.201.109@tcp129 [10918.640955] LNet: Removed LNI 192.168.201.109@tcp130 [10918.641987] LNet: Removed LNI 192.168.201.109@tcp131 [10918.643058] LNet: Removed LNI 192.168.201.109@tcp132 [10918.644150] LNet: Removed LNI 192.168.201.109@tcp133 [10918.645218] LNet: Removed LNI 192.168.201.109@tcp134 [10918.652413] LNet: Removed LNI 192.168.201.109@tcp135 [10918.910438] Key type .llcrypt unregistered [10918.911462] Key type ._llcrypt unregistered [10925.986926] Key type ._llcrypt registered [10925.988857] Key type .llcrypt registered [10926.024857] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [10931.979745] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10932.315624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10932.321629] alg: No test for adler32 (adler32-zlib) [10933.171203] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [10933.261400] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10934.832107] Key type lgssc registered [10935.177164] Lustre: Echo OBD driver; http://www.lustre.org/ [10946.242101] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:36:01 (1786196161) [10951.919352] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10952.938870] Key type lgssc unregistered [10953.065422] LNet: 471897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10953.069445] LNetError: 471897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10953.080302] LNet: Removed LNI 192.168.201.109@tcp [10953.376196] Key type .llcrypt unregistered [10953.377154] Key type ._llcrypt unregistered [10959.694815] Key type ._llcrypt registered [10959.695759] Key type .llcrypt registered [10959.730638] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10960.135891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10960.146624] alg: No test for adler32 (adler32-zlib) [10960.998535] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [10961.085065] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10962.664135] Key type lgssc registered [10963.004263] Lustre: Echo OBD driver; http://www.lustre.org/ [10965.999708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10966.006393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.085426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10967.094654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10967.132035] Lustre: lustre-MDT0000: new disk, initializing [10967.154447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10967.161402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10968.464425] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10972.431441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10972.473057] Lustre: 473087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10972.501307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10972.503028] Lustre: Skipped 1 previous similar message [10972.538024] Lustre: lustre-MDT0001: new disk, initializing [10972.574929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10972.587141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10972.590981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10973.912373] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10977.467412] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10979.211362] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10981.478137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10981.574350] Lustre: lustre-OST0000: new disk, initializing [10981.575736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10981.578707] Lustre: 475050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10981.605391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10982.775933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10982.780987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10982.814758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10983.401410] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10986.907050] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10987.992289] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [10989.523714] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10989.534256] LNet: Added LNI 192.168.201.109@tcp10 [8/256/0/180] [10991.151845] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10991.161610] LNet: Added LNI 192.168.201.109@tcp11 [8/256/0/180] [10992.393768] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10992.406185] LNet: Added LNI 192.168.201.109@tcp12 [8/256/0/180] [10993.664732] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10993.676091] LNet: Added LNI 192.168.201.109@tcp13 [8/256/0/180] [10994.962995] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10994.971994] LNet: Added LNI 192.168.201.109@tcp14 [8/256/0/180] [10996.216884] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10996.226938] LNet: Added LNI 192.168.201.109@tcp15 [8/256/0/180] [10997.518194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10997.528559] LNet: Added LNI 192.168.201.109@tcp16 [8/256/0/180] [10998.904537] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10998.913966] LNet: Added LNI 192.168.201.109@tcp17 [8/256/0/180] [11000.323558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [11000.332911] LNet: Added LNI 192.168.201.109@tcp18 [8/256/0/180] [11001.922811] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [11001.935547] LNet: Added LNI 192.168.201.109@tcp19 [8/256/0/180] [11003.521403] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [11003.530894] LNet: Added LNI 192.168.201.109@tcp20 [8/256/0/180] [11005.190968] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [11005.201456] LNet: Added LNI 192.168.201.109@tcp21 [8/256/0/180] [11006.843576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [11006.852904] LNet: Added LNI 192.168.201.109@tcp22 [8/256/0/180] [11008.557210] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [11008.568769] LNet: Added LNI 192.168.201.109@tcp23 [8/256/0/180] [11010.338559] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [11010.347971] LNet: Added LNI 192.168.201.109@tcp24 [8/256/0/180] [11012.051465] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [11012.060820] LNet: Added LNI 192.168.201.109@tcp25 [8/256/0/180] [11013.733092] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [11013.742951] LNet: Added LNI 192.168.201.109@tcp26 [8/256/0/180] [11015.541890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [11015.551747] LNet: Added LNI 192.168.201.109@tcp27 [8/256/0/180] [11017.267707] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [11017.277888] LNet: Added LNI 192.168.201.109@tcp28 [8/256/0/180] [11019.163229] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [11019.173513] LNet: Added LNI 192.168.201.109@tcp29 [8/256/0/180] [11021.004869] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [11021.013903] LNet: Added LNI 192.168.201.109@tcp30 [8/256/0/180] [11022.870652] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [11022.882118] LNet: Added LNI 192.168.201.109@tcp31 [8/256/0/180] [11024.824474] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [11024.836146] LNet: Added LNI 192.168.201.109@tcp32 [8/256/0/180] [11026.931938] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [11026.943251] LNet: Added LNI 192.168.201.109@tcp33 [8/256/0/180] [11028.996962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [11029.009019] LNet: Added LNI 192.168.201.109@tcp34 [8/256/0/180] [11031.310236] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [11031.323637] LNet: Added LNI 192.168.201.109@tcp35 [8/256/0/180] [11033.570137] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [11033.582758] LNet: Added LNI 192.168.201.109@tcp36 [8/256/0/180] [11035.812812] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [11035.823426] LNet: Added LNI 192.168.201.109@tcp37 [8/256/0/180] [11037.994517] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [11038.004417] LNet: Added LNI 192.168.201.109@tcp38 [8/256/0/180] [11040.076194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [11040.086974] LNet: Added LNI 192.168.201.109@tcp39 [8/256/0/180] [11042.305920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [11042.315204] LNet: Added LNI 192.168.201.109@tcp40 [8/256/0/180] [11044.369659] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [11044.380798] LNet: Added LNI 192.168.201.109@tcp41 [8/256/0/180] [11046.459493] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [11046.468974] LNet: Added LNI 192.168.201.109@tcp42 [8/256/0/180] [11048.604135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [11048.614102] LNet: Added LNI 192.168.201.109@tcp43 [8/256/0/180] [11050.858835] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [11050.870066] LNet: Added LNI 192.168.201.109@tcp44 [8/256/0/180] [11053.231213] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [11053.241207] LNet: Added LNI 192.168.201.109@tcp45 [8/256/0/180] [11055.656802] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [11055.667540] LNet: Added LNI 192.168.201.109@tcp46 [8/256/0/180] [11058.355541] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [11058.366656] LNet: Added LNI 192.168.201.109@tcp47 [8/256/0/180] [11060.985679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [11060.994629] LNet: Added LNI 192.168.201.109@tcp48 [8/256/0/180] [11063.483522] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [11063.495703] LNet: Added LNI 192.168.201.109@tcp49 [8/256/0/180] [11066.029328] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [11066.039832] LNet: Added LNI 192.168.201.109@tcp50 [8/256/0/180] [11075.042708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11075.043251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.044654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11075.049786] Lustre: Skipped 2 previous similar messages [11079.326836] Lustre: server umount lustre-MDT0000 complete [11080.503067] LustreError: 473077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196296 with bad export cookie 2064975198395587248 [11080.506067] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11080.506194] LustreError: 473077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11080.621415] Lustre: server umount lustre-MDT0001 complete [11087.519147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11090.358102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11090.400273] LustreError: 472358:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000876f18b3 x1872962441525760/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11090.506719] LustreError: 475044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11090.511843] LustreError: 475044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11090.522521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11091.929836] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11094.884569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11094.991781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.036041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11096.348149] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11098.546359] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11100.134723] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11100.338774] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11101.136159] Lustre: 472362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786196300/real 1786196300] req@00000000efefd963 x1872962441525504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786196316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11105.254344] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11107.296115] Lustre: 472362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786196306/real 1786196306] req@0000000067e584eb x1872962441525632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786196322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11110.371392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.372741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11110.375748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11110.379260] Lustre: Skipped 3 previous similar messages [11113.551933] Lustre: server umount lustre-OST0000 complete [11115.490081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11115.490174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.490656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.496861] Lustre: Skipped 2 previous similar messages [11120.609130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11120.610876] Lustre: Skipped 2 previous similar messages [11120.855861] Lustre: server umount lustre-MDT0000 complete [11122.066081] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196337 with bad export cookie 2064975198395589243 [11122.069039] LustreError: 473076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11122.069558] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11122.177089] Lustre: server umount lustre-MDT0001 complete [11125.695753] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11126.729518] Key type lgssc unregistered [11126.890780] LNet: 502014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11126.893657] LNetError: 502014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11126.895966] LNet: Removed LNI 192.168.201.109@tcp [11126.897152] LNet: Removed LNI 192.168.201.109@tcp10 [11126.898538] LNet: Removed LNI 192.168.201.109@tcp11 [11126.899755] LNet: Removed LNI 192.168.201.109@tcp12 [11126.900912] LNet: Removed LNI 192.168.201.109@tcp13 [11126.902106] LNet: Removed LNI 192.168.201.109@tcp14 [11126.903170] LNet: Removed LNI 192.168.201.109@tcp15 [11126.904322] LNet: Removed LNI 192.168.201.109@tcp16 [11126.905497] LNet: Removed LNI 192.168.201.109@tcp17 [11126.906696] LNet: Removed LNI 192.168.201.109@tcp18 [11126.907967] LNet: Removed LNI 192.168.201.109@tcp19 [11126.910142] LNet: Removed LNI 192.168.201.109@tcp20 [11126.912175] LNet: Removed LNI 192.168.201.109@tcp21 [11126.913870] LNet: Removed LNI 192.168.201.109@tcp22 [11126.915149] LNet: Removed LNI 192.168.201.109@tcp23 [11126.916321] LNet: Removed LNI 192.168.201.109@tcp24 [11126.917660] LNet: Removed LNI 192.168.201.109@tcp25 [11126.919450] LNet: Removed LNI 192.168.201.109@tcp26 [11126.921584] LNet: Removed LNI 192.168.201.109@tcp27 [11126.922827] LNet: Removed LNI 192.168.201.109@tcp28 [11126.923936] LNet: Removed LNI 192.168.201.109@tcp29 [11126.925217] LNet: Removed LNI 192.168.201.109@tcp30 [11126.926407] LNet: Removed LNI 192.168.201.109@tcp31 [11126.927803] LNet: Removed LNI 192.168.201.109@tcp32 [11126.929108] LNet: Removed LNI 192.168.201.109@tcp33 [11126.930280] LNet: Removed LNI 192.168.201.109@tcp34 [11126.931482] LNet: Removed LNI 192.168.201.109@tcp35 [11126.932642] LNet: Removed LNI 192.168.201.109@tcp36 [11126.933770] LNet: Removed LNI 192.168.201.109@tcp37 [11126.934942] LNet: Removed LNI 192.168.201.109@tcp38 [11126.936150] LNet: Removed LNI 192.168.201.109@tcp39 [11126.937425] LNet: Removed LNI 192.168.201.109@tcp40 [11126.938547] LNet: Removed LNI 192.168.201.109@tcp41 [11126.940885] LNet: Removed LNI 192.168.201.109@tcp42 [11126.943340] LNet: Removed LNI 192.168.201.109@tcp43 [11126.944538] LNet: Removed LNI 192.168.201.109@tcp44 [11126.945872] LNet: Removed LNI 192.168.201.109@tcp45 [11126.947112] LNet: Removed LNI 192.168.201.109@tcp46 [11126.948386] LNet: Removed LNI 192.168.201.109@tcp47 [11126.949534] LNet: Removed LNI 192.168.201.109@tcp48 [11126.950741] LNet: Removed LNI 192.168.201.109@tcp49 [11126.957511] LNet: Removed LNI 192.168.201.109@tcp50 [11127.285811] Key type .llcrypt unregistered [11127.287567] Key type ._llcrypt unregistered [11133.600201] Key type ._llcrypt registered [11133.601157] Key type .llcrypt registered [11133.634839] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [11139.391978] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11139.706260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11139.733074] alg: No test for adler32 (adler32-zlib) [11140.584759] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [11140.681684] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11142.264095] Key type lgssc registered [11142.609603] Lustre: Echo OBD driver; http://www.lustre.org/ [11153.065375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11153.071319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.152663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11154.161966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11154.200903] Lustre: lustre-MDT0000: new disk, initializing [11154.221741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11154.228564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11155.504135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11159.512843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.555517] Lustre: 505186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11159.579407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11159.580937] Lustre: Skipped 1 previous similar message [11159.616344] Lustre: lustre-MDT0001: new disk, initializing [11159.646845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11159.658316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11159.661148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11160.986287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11164.666776] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11166.324417] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11168.753274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11168.821805] Lustre: *** cfs_fail_loc=90f, val=0*** [11168.848796] Lustre: lustre-OST0000: new disk, initializing [11168.850455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11168.852816] Lustre: 507200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11168.879189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11170.740242] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11174.342190] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11176.442828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11176.446476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11176.488561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11181.538542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11181.540181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.540870] LustreError: Skipped 1 previous similar message [11181.546816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.549756] Lustre: Skipped 1 previous similar message [11182.805381] Lustre: server umount lustre-OST0000 complete [11185.120825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11185.122919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11185.125943] Lustre: Skipped 1 previous similar message [11185.127671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11186.657352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11186.657971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11190.244917] Lustre: server umount lustre-MDT0000 complete [11191.496646] LustreError: 505178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196407 with bad export cookie 4876045492130497917 [11191.501754] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.652861] Lustre: server umount lustre-MDT0001 complete [11195.182063] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11196.273445] Key type lgssc unregistered [11196.405396] LNet: 509103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11196.408133] LNetError: 509103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11196.414342] LNet: Removed LNI 192.168.201.109@tcp [11196.729590] Key type .llcrypt unregistered [11196.730662] Key type ._llcrypt unregistered [11203.979171] Key type ._llcrypt registered [11203.980195] Key type .llcrypt registered [11204.016619] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11204.389900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11204.405436] alg: No test for adler32 (adler32-zlib) [11205.251179] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [11205.333821] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11206.912151] Key type lgssc registered [11207.247921] Lustre: Echo OBD driver; http://www.lustre.org/ [11209.697261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11209.704884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11210.851075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11212.088365] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11215.367548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.537076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11217.127608] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11219.776194] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11221.802609] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11224.105714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11224.223045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11226.304380] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11228.833597] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11243.567845] loop0: detected capacity change from 0 to 204800000 [11243.642126] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11243.647037] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11243.650882] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11245.963738] loop0: detected capacity change from 0 to 204800000 [11246.028716] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11246.032760] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11246.036583] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11247.886552] loop0: detected capacity change from 0 to 204800000 [11250.406198] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11250.474915] Lustre: fs15246-OST0000: new disk, initializing [11250.477266] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11250.480111] Lustre: 513961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11250.510197] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11252.967804] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11257.573868] loop1: detected capacity change from 0 to 204800000 [11260.223991] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.277304] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11260.313398] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11260.363356] Lustre: fs15246-MDT0000: new disk, initializing [11260.428336] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11260.440423] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11260.494449] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11261.536904] Lustre: 515084:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000064401a43 x1872962697330560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786196493 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11261.845564] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11270.624332] LustreError: 515084:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [11270.631389] LustreError: 515084:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [11270.637936] LustreError: 515084:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [11272.172061] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.180598] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11276.211563] Lustre: server umount fs15246-MDT0000 complete [11277.287096] LustreError: 512836:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.296074] LustreError: 512836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11280.669654] Lustre: server umount fs15246-OST0000 complete [11287.009404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11287.013413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.020872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11291.248424] Lustre: server umount lustre-OST0000 complete [11297.763140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.763210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11297.764378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11297.764384] Lustre: Skipped 1 previous similar message [11297.769038] Lustre: Skipped 2 previous similar messages [11298.929493] Lustre: server umount lustre-MDT0000 complete [11300.438113] LustreError: 510330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196516 with bad export cookie 5116141138197965452 [11300.442201] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11300.444879] LustreError: 510330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11300.598783] Lustre: server umount lustre-MDT0001 complete [11304.856548] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11306.185455] Key type lgssc unregistered [11306.335425] LNet: 518179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11306.338419] LNetError: 518179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11306.351316] LNet: Removed LNI 192.168.201.109@tcp [11306.803089] Key type .llcrypt unregistered [11306.805183] Key type ._llcrypt unregistered [11314.526421] Key type ._llcrypt registered [11314.527774] Key type .llcrypt registered [11314.565417] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11314.900070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11314.907259] alg: No test for adler32 (adler32-zlib) [11315.783041] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [11315.878544] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11317.456187] Key type lgssc registered [11317.814047] Lustre: Echo OBD driver; http://www.lustre.org/ [11333.926233] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [11338.860417] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11355.398364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11355.406383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.513227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11356.523680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11356.565777] Lustre: lustre-MDT0000: new disk, initializing [11356.592757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11356.599225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11358.271372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11362.964859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.021344] Lustre: 522614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11363.051789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11363.053351] Lustre: Skipped 1 previous similar message [11363.092140] Lustre: lustre-MDT0001: new disk, initializing [11363.128231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.142713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11363.145571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11364.433824] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11368.684743] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11370.492466] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11373.110137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11373.223279] Lustre: lustre-OST0000: new disk, initializing [11373.224758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11373.227068] Lustre: 524583:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11373.261580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11375.189158] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11375.357539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11375.361738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11375.403640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11378.817632] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11380.109628] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11380.175716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11381.461662] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11381.526923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11385.826320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11385.827835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11385.828564] LustreError: Skipped 1 previous similar message [11385.832178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11385.835479] Lustre: Skipped 1 previous similar message [11388.238799] Lustre: server umount lustre-OST0000 complete [11390.945688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11390.946507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11390.950848] Lustre: Skipped 2 previous similar messages [11395.545975] Lustre: server umount lustre-MDT0000 complete [11396.714112] LustreError: 522607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786196612 with bad export cookie 12610062313708625665 [11396.716724] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11396.717032] LustreError: 522607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11396.830042] Lustre: server umount lustre-MDT0001 complete [11403.985103] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [11407.378716] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11417.206243] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11098 sec ========= 09:43:52 (1786196632) [11417.696370] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:43:53 (1786196633) === [11418.740547] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:43:54 (1786196634) === [11435.192706] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11436.489915] Key type lgssc unregistered [11436.644340] LNet: 531216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11436.646953] LNetError: 531216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11436.654371] LNet: Removed LNI 192.168.201.109@tcp [11436.968185] Key type .llcrypt unregistered [11436.969099] Key type ._llcrypt unregistered