[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519615630 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003146] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006011] kvm-guest: setup PV IPIs [ 0.008861] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011017] pid_max: default: 32768 minimum: 301 [ 0.013155] LSM: Security Framework initializing [ 0.014061] Yama: becoming mindful. [ 0.016005] SELinux: Initializing. [ 0.017081] *** VALIDATE selinux *** [ 0.025874] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030754] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035009] *** VALIDATE tmpfs *** [ 0.036344] *** VALIDATE proc *** [ 0.037219] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.040237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043034] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.046885] debug: unmapping init [mem 0xffffffff8f059000-0xffffffff8f060fff] [ 0.048832] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049779] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050027] ... version: 2 [ 0.051018] ... bit width: 48 [ 0.052012] ... generic registers: 4 [ 0.053016] ... value mask: 0000ffffffffffff [ 0.054024] ... max period: 00007fffffffffff [ 0.055029] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057347] rcu: Hierarchical SRCU implementation. [ 0.059584] smp: Bringing up secondary CPUs ... [ 0.060649] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.064411] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.158026] node 0 deferred pages initialised in 87ms [ 0.162117] devtmpfs: initialized [ 0.163352] x86/mm: Memory block size: 128MB [ 0.167338] gcov: version magic: 0x41383552 [ 0.169313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.173105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.175334] pinctrl core: initialized pinctrl subsystem [ 0.178190] [ 0.178829] ************************************************************* [ 0.181016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183013] ** ** [ 0.185014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.188014] ** ** [ 0.191017] ** This means that this kernel is built to expose internal ** [ 0.193017] ** IOMMU data structures, which may compromise security on ** [ 0.195014] ** your system. ** [ 0.198014] ** ** [ 0.200016] ** If you see this message and you are not debugging the ** [ 0.202016] ** kernel, report this immediately to your vendor! ** [ 0.204017] ** ** [ 0.207016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209014] ************************************************************* [ 0.212808] NET: Registered protocol family 16 [ 0.214470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222083] cpuidle: using governor menu [ 0.223627] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224611] PCI: Using configuration type 1 for base access [ 0.226144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.237072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244101] cryptd: max_cpu_qlen set to 1000 [ 0.247247] ACPI: Added _OSI(Module Device) [ 0.249016] ACPI: Added _OSI(Processor Device) [ 0.250015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.252013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.257059] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262558] ACPI: Interpreter enabled [ 0.264071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.266015] ACPI: Using IOAPIC for interrupt routing [ 0.268128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.273392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.284280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.293076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298000] acpiphp: Slot [2] registered [ 0.299122] acpiphp: Slot [5] registered [ 0.301119] acpiphp: Slot [6] registered [ 0.302112] acpiphp: Slot [7] registered [ 0.304116] acpiphp: Slot [8] registered [ 0.305186] acpiphp: Slot [9] registered [ 0.306155] acpiphp: Slot [10] registered [ 0.308133] acpiphp: Slot [3] registered [ 0.310187] acpiphp: Slot [4] registered [ 0.311136] acpiphp: Slot [11] registered [ 0.313103] acpiphp: Slot [12] registered [ 0.314089] acpiphp: Slot [13] registered [ 0.316141] acpiphp: Slot [14] registered [ 0.318103] acpiphp: Slot [15] registered [ 0.319099] acpiphp: Slot [16] registered [ 0.321119] acpiphp: Slot [17] registered [ 0.322125] acpiphp: Slot [18] registered [ 0.324181] acpiphp: Slot [19] registered [ 0.326114] acpiphp: Slot [20] registered [ 0.328076] acpiphp: Slot [21] registered [ 0.329093] acpiphp: Slot [22] registered [ 0.331089] acpiphp: Slot [23] registered [ 0.332113] acpiphp: Slot [24] registered [ 0.334092] acpiphp: Slot [25] registered [ 0.335098] acpiphp: Slot [26] registered [ 0.337093] acpiphp: Slot [27] registered [ 0.338081] acpiphp: Slot [28] registered [ 0.340116] acpiphp: Slot [29] registered [ 0.342092] acpiphp: Slot [30] registered [ 0.344070] acpiphp: Slot [31] registered [ 0.345058] PCI host bridge to bus 0000:00 [ 0.346011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362873] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.366231] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376764] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.381553] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.398046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400824] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.419020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.424019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431082] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441067] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.488775] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.504021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.528021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.543015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.550018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.574764] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.602028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.612318] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.641015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.652663] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.686882] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.689443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.691322] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.694379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697331] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.703100] iommu: Default domain type: Passthrough [ 0.704448] SCSI subsystem initialized [ 0.705197] ACPI: bus type USB registered [ 0.706102] usbcore: registered new interface driver usbfs [ 0.707135] usbcore: registered new interface driver hub [ 0.708171] usbcore: registered new device driver usb [ 0.710186] pps_core: LinuxPPS API ver. 1 registered [ 0.712012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715066] PTP clock support registered [ 0.717164] EDAC MC: Ver: 3.0.0 [ 0.719749] PCI: Using ACPI for IRQ routing [ 0.722133] NetLabel: Initializing [ 0.724011] NetLabel: domain hash size = 128 [ 0.726014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.728103] NetLabel: unlabeled traffic allowed by default [ 0.730457] vgaarb: loaded [ 0.731363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.732030] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.740000] clocksource: Switched to clocksource kvm-clock [ 0.851494] VFS: Disk quotas dquot_6.6.0 [ 0.853605] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.856544] *** VALIDATE ramfs *** [ 0.857929] *** VALIDATE hugetlbfs *** [ 0.859751] pnp: PnP ACPI init [ 0.862369] pnp: PnP ACPI: found 6 devices [ 0.878240] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.882212] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.883836] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.885883] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.888501] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.891271] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.894447] NET: Registered protocol family 2 [ 0.897123] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.902235] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.906342] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.911505] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.914633] TCP: Hash tables configured (established 65536 bind 65536) [ 0.918252] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.921728] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928972] NET: Registered protocol family 1 [ 0.931385] RPC: Registered named UNIX socket transport module. [ 0.933403] RPC: Registered udp transport module. [ 0.935350] RPC: Registered tcp transport module. [ 0.936868] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939375] NET: Registered protocol family 44 [ 0.941167] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943554] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.945798] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.948393] PCI: CLS 0 bytes, default 64 [ 0.950794] Unpacking initramfs... [ 2.384040] debug: unmapping init [mem 0xffff99f73cc54000-0xffff99f73ffbffff] [ 2.387280] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388954] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.390971] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.874292] Initialise system trusted keyrings [ 2.875851] Key type blacklist registered [ 2.878177] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.887688] zbud: loaded [ 2.891160] *** VALIDATE nfs *** [ 2.892536] *** VALIDATE nfs4 *** [ 2.894171] pstore: using deflate compression [ 2.898251] Platform Keyring initialized [ 3.004125] NET: Registered protocol family 38 [ 3.005965] Key type asymmetric registered [ 3.007188] Asymmetric key parser 'x509' registered [ 3.009050] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.011905] io scheduler mq-deadline registered [ 3.013591] io scheduler kyber registered [ 3.015381] io scheduler bfq registered [ 3.017372] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.019706] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.022095] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024755] ACPI: Power Button [PWRF] [ 3.029983] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.036357] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.049438] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.057064] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.075815] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.102933] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.132869] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.140682] Non-volatile memory driver v1.3 [ 3.142473] Linux agpgart interface v0.103 [ 3.181585] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 3.185185] vda: detected capacity change from 0 to 74149888 [ 3.205759] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.209160] vdb: detected capacity change from 0 to 1073741824 [ 3.223480] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.226454] vdc: detected capacity change from 0 to 2621440000 [ 3.241460] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.243969] vdd: detected capacity change from 0 to 2621440000 [ 3.259987] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.262453] vde: detected capacity change from 0 to 4294967296 [ 3.277197] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.280525] vdf: detected capacity change from 0 to 4294967296 [ 3.287834] libphy: Fixed MDIO Bus: probed [ 3.293498] usbcore: registered new interface driver usbserial_generic [ 3.296191] usbserial: USB Serial support registered for generic [ 3.298754] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.302502] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304055] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.306707] mousedev: PS/2 mouse device common for all mice [ 3.310076] rtc_cmos 00:05: RTC can wake from S4 [ 3.312272] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.313877] rtc_cmos 00:05: registered as rtc0 [ 3.318157] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.319732] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.320906] intel_pstate: CPU model not supported [ 3.326215] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.327504] hid: raw HID events driver (C) Jiri Kosina [ 3.330933] usbcore: registered new interface driver usbhid [ 3.333013] usbhid: USB HID core driver [ 3.334349] drop_monitor: Initializing network drop monitor service [ 3.337019] Initializing XFRM netlink socket [ 3.339246] NET: Registered protocol family 10 [ 3.342279] Segment Routing with IPv6 [ 3.343845] NET: Registered protocol family 17 [ 3.345869] mpls_gso: MPLS GSO support [ 3.352628] RAS: Correctable Errors collector initialized. [ 3.354798] AVX version of gcm_enc/dec engaged. [ 3.356665] AES CTR mode by8 optimization enabled [ 3.439086] sched_clock: Marking stable (3439038885, 0)->(4328490476, -889451591) [ 3.442744] registered taskstats version 1 [ 3.445666] Loading compiled-in X.509 certificates [ 3.447633] zswap: loaded using pool lzo/zbud [ 3.473074] Key type big_key registered [ 3.486665] Key type encrypted registered [ 3.488388] ima: No TPM chip found, activating TPM-bypass! [ 3.490466] ima: Allocated hash algorithm: sha1 [ 3.491954] ima: No architecture policies found [ 3.493430] evm: Initialising EVM extended attributes: [ 3.494940] evm: security.selinux [ 3.495709] evm: security.ima [ 3.496391] evm: security.capability [ 3.497322] evm: HMAC attrs: 0x1 [ 3.499335] rtc_cmos 00:05: setting system clock to 2026-06-27 03:14:27 UTC (1782530067) [ 3.505944] debug: unmapping init [mem 0xffffffff90003000-0xffffffff901fffff] [ 3.508934] debug: unmapping init [mem 0xffffffff8ed82000-0xffffffff8f058fff] [ 3.518123] Write protecting the kernel read-only data: 28672k [ 3.521179] debug: unmapping init [mem 0xffffffff8d403000-0xffffffff8d5fffff] [ 3.524213] debug: unmapping init [mem 0xffffffff8dd14000-0xffffffff8ddfffff] [ 3.559149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.567786] systemd[1]: Detected virtualization kvm. [ 3.569775] systemd[1]: Detected architecture x86-64. [ 3.572203] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.598069] systemd[1]: No hostname configured. [ 3.599419] systemd[1]: Set hostname to . [ 3.601934] random: systemd: uninitialized urandom read (16 bytes read) [ 3.604407] systemd[1]: Initializing machine ID from random generator. [ 3.727794] random: systemd: uninitialized urandom read (16 bytes read) [ 3.730489] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.734976] random: systemd: uninitialized urandom read (16 bytes read) [ 3.738195] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.745424] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.311340] device-mapper: uevent: version 1.0.3 [ 4.313361] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.860215] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.056415] virtio_net virtio0 ens2: renamed from eth0 [ 5.067248] scsi host0: ata_piix [ 5.104800] scsi host1: ata_piix [ 5.106609] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.109289] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.766782] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.017902] random: crng init done [ 10.019320] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.330779] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.430738] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.656711] SELinux: Disabled at runtime. [ 11.720621] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.729549] systemd[1]: Detected virtualization kvm. [ 11.731238] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.168984] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.171168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.175325] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.179344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.182599] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.188327] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.211311] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ 12.258585] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.571261] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.840279] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.852600] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.979262] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.987686] EDAC sbridge: Ver: 1.1.2 [ 14.307263] Key type dns_resolver registered [ 14.594984] NFS: Registering the id_resolver key type [ 14.596972] Key type id_resolver registered [ 14.598526] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 34.067195] hrtimer: interrupt took 2404674 ns [ 73.137581] libcfs: loading out-of-tree module taints kernel. [ 73.158929] Key type ._llcrypt registered [ 73.160807] Key type .llcrypt registered [ 73.260065] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 92.215588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 93.872094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.891609] alg: No test for adler32 (adler32-zlib) [ 95.513393] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 96.520036] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 98.303174] Key type lgssc registered [ 100.099713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.655643] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.818821] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 184.707031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.749753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.103317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.135197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.210534] Lustre: lustre-MDT0000: new disk, initializing [ 186.289912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.301652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.602731] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.568778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.693663] Lustre: 6526:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.753578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.759051] Lustre: Skipped 1 previous similar message [ 206.927882] Lustre: lustre-MDT0001: new disk, initializing [ 207.011458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.060374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.079506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.102442] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.271924] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.315050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.715257] Lustre: lustre-OST0000: new disk, initializing [ 229.718597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.723963] Lustre: 8461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.778917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 231.039750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.048947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.155601] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.493024] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.381962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.540971] Lustre: lustre-OST0001: new disk, initializing [ 252.546261] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.555664] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.616798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.116886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 258.129080] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 258.211869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.462982] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.267661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.454429] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.377306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.970811] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 304.329288] Lustre: DEBUG MARKER: Client: 2.17.54.99 [ 307.915653] Lustre: DEBUG MARKER: MDS: 2.17.54.99 [ 311.109588] Lustre: DEBUG MARKER: OSS: 2.17.54.99 [ 313.025521] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 23:19:35 EDT 2026 [ 333.792034] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 335.228465] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.823270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.831781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.846321] Lustre: Skipped 1 previous similar message [ 355.862341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 357.533873] Lustre: server umount lustre-MDT0000 complete [ 361.935480] LustreError: 6517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782530425 with bad export cookie 11018779861561622636 [ 361.956726] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.048272] LustreError: 10052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.048758] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.050345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.050352] Lustre: Skipped 3 previous similar messages [ 366.061204] LustreError: 10052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 366.104919] Lustre: Skipped 3 previous similar messages [ 371.167945] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.168099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.186596] LustreError: 6531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 371.194015] Lustre: Skipped 2 previous similar messages [ 372.192806] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.220228] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 376.287306] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 376.561153] Lustre: server umount lustre-MDT0001 complete [ 380.877673] Lustre: server umount lustre-OST0000 complete [ 385.518309] Lustre: server umount lustre-OST0001 complete [ 393.890060] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 406.484737] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 451.769871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.180568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.205678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.383197] Lustre: lustre-MDT0000: new disk, initializing [ 452.527857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.564367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 458.520777] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 470.885281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.018244] Lustre: 15097:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.102476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.105694] Lustre: Skipped 1 previous similar message [ 471.226267] Lustre: lustre-MDT0001: new disk, initializing [ 471.386134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.452179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 471.471253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 477.560427] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 489.247205] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.356645] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 504.293317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 504.564282] Lustre: lustre-OST0000: new disk, initializing [ 504.571175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 504.583195] Lustre: 17067:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 504.684064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.048892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 506.062399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 506.273510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 513.026251] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 525.632942] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 530.613516] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 530.808954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 535.398652] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 535.637983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.181364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.186645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.206608] LustreError: Skipped 1 previous similar message [ 542.216644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.218650] Lustre: Skipped 1 previous similar message [ 542.238784] Lustre: Skipped 3 previous similar messages [ 544.016802] Lustre: server umount lustre-OST0000 complete [ 548.320721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 548.331725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.346624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.417317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.538405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.555580] Lustre: Skipped 3 previous similar messages [ 562.144611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 562.383623] Lustre: server umount lustre-MDT0000 complete [ 562.656996] LustreError: 15956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.679584] LustreError: 15956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 566.419867] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782530630 with bad export cookie 11018779861561624169 [ 566.423346] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.438255] LustreError: 15088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 566.769235] Lustre: server umount lustre-MDT0001 complete [ 571.030832] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:23:53 (1782530633) [ 573.682557] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 575.887626] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:23:58 (1782530638) [ 585.631098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.224282] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.391113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.277931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 596.960454] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.984322] LustreError: 19480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 601.080068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.597147] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 615.639534] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 622.829528] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 632.112316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 640.331178] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 649.735828] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 671.719901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 671.733192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.741597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 671.743657] Lustre: Skipped 1 previous similar message [ 671.770821] Lustre: Skipped 1 previous similar message [ 674.714310] Lustre: server umount lustre-OST0000 complete [ 685.797351] Lustre: server umount lustre-MDT0000 complete [ 687.072149] LustreError: 20208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.116822] LustreError: 20208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 690.438042] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782530754 with bad export cookie 11018779861561625044 [ 690.452663] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.456752] LustreError: 19461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 690.850645] Lustre: server umount lustre-MDT0001 complete [ 703.307973] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 707.071127] Key type lgssc unregistered [ 707.424433] LNet: 23038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.436541] LNetError: 23038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.465263] LNet: Removed LNI 192.168.201.103@tcp [ 708.518248] Key type .llcrypt unregistered [ 708.521389] Key type ._llcrypt unregistered [ 734.559937] Key type ._llcrypt registered [ 734.568880] Key type .llcrypt registered [ 734.677476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 736.017121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.304458] alg: No test for adler32 (adler32-zlib) [ 737.760318] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 738.229369] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 740.152058] Key type lgssc registered [ 742.097880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 753.368692] loop: module loaded [ 753.454148] loop0: detected capacity change from 0 to 204800000 [ 753.697583] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.713165] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 754.137326] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 761.932430] loop0: detected capacity change from 0 to 204800000 [ 762.293389] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 762.315518] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 762.336957] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 768.852182] loop0: detected capacity change from 0 to 204800000 [ 776.106844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 776.139769] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.469191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 777.564314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 777.709519] Lustre: lustre-MDT0000: new disk, initializing [ 777.825549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.871486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 783.366694] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 793.697910] loop1: detected capacity change from 0 to 204800000 [ 800.332953] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 800.591480] Lustre: lustre-OST07c6: new disk, initializing [ 800.595796] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 800.602029] Lustre: Skipped 1 previous similar message [ 800.608614] Lustre: 26229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 800.718690] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 808.537870] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 809.024164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 809.048401] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 809.098528] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 814.435908] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 814.457230] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 814.467662] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 814.485990] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 814.501861] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 814.516270] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 831.211398] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 831.220385] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 831.224777] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 831.229357] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 831.235825] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 831.238806] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 836.469356] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 836.484074] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 836.499987] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 836.506594] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.514531] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 836.518570] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.524684] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 836.529549] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.535993] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 836.541804] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.548346] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 836.554660] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.759539] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 839.774863] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 839.783860] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 839.803589] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.820369] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 839.826047] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.833380] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 839.844732] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.862597] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 839.865867] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 839.869634] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 839.873123] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.597891] Lustre: 25214:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 844.617599] Lustre: 25214:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 844.627163] Lustre: 25214:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 844.631722] Lustre: 25214:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.636568] Lustre: 25214:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 844.643669] Lustre: 25214:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.650048] Lustre: 25214:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 844.659425] Lustre: 25214:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.667985] Lustre: 25214:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 844.677215] Lustre: 25214:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 844.681983] Lustre: 25214:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 844.689697] Lustre: 25214:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 852.786838] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 852.794912] Lustre: 25215:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 852.800259] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 852.805490] Lustre: 25215:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 852.813373] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 852.817831] Lustre: 25215:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 852.830631] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 852.845562] Lustre: 25215:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 852.865400] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 852.888818] Lustre: 25215:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 852.898061] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 852.908537] Lustre: 25215:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 869.349238] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 869.377163] Lustre: 26604:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 869.393476] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 869.406645] Lustre: 26604:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 869.421590] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 869.433335] Lustre: 26604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 869.439233] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 869.457307] Lustre: 26604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 869.471744] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 869.483133] Lustre: 26604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 869.491819] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 869.498310] Lustre: 26604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 906.209178] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.218889] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.240205] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 911.328983] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 916.450528] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 920.031213] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 920.186976] Lustre: server umount lustre-OST07c6 complete [ 932.008440] Lustre: server umount lustre-MDT0000 complete [ 951.334257] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 955.159630] Key type lgssc unregistered [ 955.643732] LNet: 28787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.651667] LNetError: 28787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.674289] LNet: Removed LNI 192.168.201.103@tcp [ 956.746249] Key type .llcrypt unregistered [ 956.753255] Key type ._llcrypt unregistered [ 978.127926] Key type ._llcrypt registered [ 978.134298] Key type .llcrypt registered [ 978.344818] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 980.063901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.238215] alg: No test for adler32 (adler32-zlib) [ 981.544994] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 981.908904] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 983.784617] Key type lgssc registered [ 985.314431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.161970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 995.191312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.628661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.456249] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1006.049451] LustreError: 30060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.064110] LustreError: 30060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1010.420069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.754465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1015.039414] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1023.025523] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.495723] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1037.065404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.548729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.672169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1045.090413] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1052.781307] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1071.588142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.592177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1071.620632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1071.637751] Lustre: Skipped 1 previous similar message [ 1076.725334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1076.734818] Lustre: Skipped 1 previous similar message [ 1077.079499] Lustre: server umount lustre-OST0000 complete [ 1081.827611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.841409] Lustre: Skipped 1 previous similar message [ 1081.848269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.855208] Lustre: Skipped 1 previous similar message [ 1086.947115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1086.949854] Lustre: Skipped 1 previous similar message [ 1088.109050] Lustre: server umount lustre-MDT0000 complete [ 1092.069168] LustreError: 30055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.091018] LustreError: 30055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1093.294857] LustreError: 30041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531157 with bad export cookie 11142338799620638731 [ 1093.308895] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1093.363156] LustreError: 30041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1093.892144] Lustre: server umount lustre-MDT0001 complete [ 1108.778466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1111.879306] Key type lgssc unregistered [ 1112.136785] LNet: 33799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.141869] LNetError: 33799:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.195184] LNet: Removed LNI 192.168.201.103@tcp [ 1114.290665] Key type .llcrypt unregistered [ 1114.293537] Key type ._llcrypt unregistered [ 1134.201201] Key type ._llcrypt registered [ 1134.206450] Key type .llcrypt registered [ 1134.356487] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1135.496474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1135.601893] alg: No test for adler32 (adler32-zlib) [ 1136.746577] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1137.010350] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1138.735862] Key type lgssc registered [ 1139.874824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1148.548041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1148.563953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.298418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.006331] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1160.675067] LustreError: 35070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.693791] LustreError: 35070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1165.447436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.811290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.649704] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1178.515118] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1185.145310] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1192.732133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.008154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.084755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1200.395363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1209.470922] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1220.586652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1220.596169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.606645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.089882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1221.100814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1221.134729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1221.144272] Lustre: Skipped 1 previous similar message [ 1226.214997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1226.229380] Lustre: Skipped 1 previous similar message [ 1226.697898] Lustre: server umount lustre-OST0000 complete [ 1231.335204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.347384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1231.371231] Lustre: Skipped 1 previous similar message [ 1231.391032] Lustre: Skipped 1 previous similar message [ 1236.450109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.844209] Lustre: server umount lustre-MDT0000 complete [ 1240.985151] LustreError: 35050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531304 with bad export cookie 6161931736651973217 [ 1240.996957] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1241.420688] Lustre: server umount lustre-MDT0001 complete [ 1255.488599] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1258.285339] Key type lgssc unregistered [ 1258.669771] LNet: 38605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.675142] LNetError: 38605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.691276] LNet: Removed LNI 192.168.201.103@tcp [ 1259.640975] Key type .llcrypt unregistered [ 1259.642967] Key type ._llcrypt unregistered [ 1280.996931] Key type ._llcrypt registered [ 1281.000960] Key type .llcrypt registered [ 1281.119990] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1282.393251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1282.438615] alg: No test for adler32 (adler32-zlib) [ 1283.598133] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1283.899369] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1285.567241] Key type lgssc registered [ 1287.017552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1294.691259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1294.709622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.075712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1300.312101] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1306.593594] LustreError: 40017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1306.615471] LustreError: 40017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1309.221857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.575489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1313.762652] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1321.807317] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1328.069793] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1334.889069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1335.306706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.495445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1342.396958] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1349.331530] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.844285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.861353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.865093] Lustre: Skipped 2 previous similar messages [ 1359.879825] Lustre: Skipped 1 previous similar message [ 1362.219580] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1364.966238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1364.970685] Lustre: Skipped 1 previous similar message [ 1365.773980] Lustre: server umount lustre-MDT0000 complete [ 1367.298436] LustreError: 40014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.832168] LustreError: 39998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531433 with bad export cookie 13341628857493691171 [ 1369.848517] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1370.082363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.082928] LustreError: 40012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.098354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1370.098363] Lustre: Skipped 1 previous similar message [ 1372.415735] LustreError: 41301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.113627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1374.119207] Lustre: Skipped 1 previous similar message [ 1376.236550] Lustre: server umount lustre-MDT0001 complete [ 1391.712704] Lustre: server umount lustre-OST0000 complete [ 1410.934375] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1413.541940] Key type lgssc unregistered [ 1413.788495] LNet: 43879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.796701] LNetError: 43879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.815680] LNet: Removed LNI 192.168.201.103@tcp [ 1414.824565] Key type .llcrypt unregistered [ 1414.827451] Key type ._llcrypt unregistered [ 1443.159823] Key type ._llcrypt registered [ 1443.162570] Key type .llcrypt registered [ 1443.266098] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1444.875867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.897720] alg: No test for adler32 (adler32-zlib) [ 1446.036553] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1446.282703] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1448.007549] Key type lgssc registered [ 1449.247066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.778306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.815403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.351384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.957878] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1469.921491] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.945064] LustreError: 45296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1473.252928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.604226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1478.862568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1487.069335] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.681981] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1500.794723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.160849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.248310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1508.602884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1517.761360] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1524.718488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1524.726310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.741703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.806256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.807279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1528.808131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.808136] Lustre: Skipped 1 previous similar message [ 1530.113239] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1530.120366] Lustre: Skipped 1 previous similar message [ 1530.632719] Lustre: server umount lustre-OST0000 complete [ 1548.104434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.111224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1548.132094] Lustre: Skipped 1 previous similar message [ 1548.157579] Lustre: Skipped 1 previous similar message [ 1554.406469] LustreError: 45292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.424597] LustreError: 45292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1554.598348] Lustre: server umount lustre-MDT0000 complete [ 1558.150804] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531622 with bad export cookie 4177904792254559930 [ 1558.155772] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1558.161783] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1564.504641] Lustre: server umount lustre-MDT0001 complete [ 1574.048499] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1576.476943] Key type lgssc unregistered [ 1576.727506] LNet: 49004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.735954] LNetError: 49004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.764836] LNet: Removed LNI 192.168.201.103@tcp [ 1577.734511] Key type .llcrypt unregistered [ 1577.737326] Key type ._llcrypt unregistered [ 1602.177286] Key type ._llcrypt registered [ 1602.183666] Key type .llcrypt registered [ 1602.350020] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1603.325733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.376899] alg: No test for adler32 (adler32-zlib) [ 1604.357838] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1604.569725] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1606.239111] Key type lgssc registered [ 1607.273895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.539938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.569784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.126618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.230205] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1626.593907] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.616485] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1628.863096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.173383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1633.679434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1640.742748] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1646.272908] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1653.280169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.845269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.835664] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1662.025423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1668.605512] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1675.239455] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1678.472805] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1690.790166] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-2623h42m32s [ 1692.153835] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1695.714748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.722357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.734415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.802880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.811066] Lustre: Skipped 4 previous similar messages [ 1698.113406] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1698.117148] Lustre: Skipped 1 previous similar message [ 1700.257821] Lustre: server umount lustre-MDT0000 complete [ 1702.886195] LustreError: 50415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.918350] LustreError: 50415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1703.318500] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1708.984774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.028787] LustreError: 51659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1709.045616] LustreError: 51659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1709.107965] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.290075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.340126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1713.270467] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1715.707305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1716.346755] Lustre: DEBUG MARKER: Wait for df (36585) ... [ 1717.716764] Lustre: DEBUG MARKER: done [ 1721.323523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.335179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.341859] Lustre: Skipped 2 previous similar messages [ 1721.352394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.901804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1724.906382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.922703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1725.930971] Lustre: Skipped 2 previous similar messages [ 1727.487668] Lustre: server umount lustre-OST0000 complete [ 1731.041915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1731.053456] Lustre: Skipped 1 previous similar message [ 1732.063133] Lustre: 49731:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1782531795/real 0] req@00000000df1a475c x1869119926364672/t0(0) o41->lustre-MDT0000-osp-MDT0001@127.0.0.2@tcp:24/4 lens 224/368 e 0 to 1 dl 1782531796 ref 1 fl Rpc:XQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1736.883721] Lustre: server umount lustre-MDT0000 complete [ 1737.121305] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.139505] LustreError: 50420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1740.056487] LustreError: 50400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531804 with bad export cookie 1153766284822520229 [ 1740.060763] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.070155] LustreError: 50400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1740.398678] Lustre: server umount lustre-MDT0001 complete [ 1750.035485] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1752.565197] Key type lgssc unregistered [ 1752.829993] LNet: 55347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.837974] LNetError: 55347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.897499] LNet: Removed LNI 192.168.201.103@tcp [ 1754.798202] Key type .llcrypt unregistered [ 1754.801322] Key type ._llcrypt unregistered [ 1787.610853] Key type ._llcrypt registered [ 1787.612789] Key type .llcrypt registered [ 1787.693695] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1788.735957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.763341] alg: No test for adler32 (adler32-zlib) [ 1789.797408] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1789.973475] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1791.647778] Key type lgssc registered [ 1792.763074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.267520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.286750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.535424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1800.569502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1800.693502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.625773] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1811.645618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.732791] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.787834] Lustre: 57537:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1812.090446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.026502] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1822.393162] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1827.349911] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1833.035603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1833.196614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1833.351285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.394291] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1841.774714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1843.686655] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1849.721086] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-2623h45m12s [ 1850.994593] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1856.811622] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1858.921855] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1859.045286] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1879.326678] Lustre: lustre-MDT0000: Client 0aefe8bb-2de2-4168-a96d-1898db78f496 (at 192.168.201.3@tcp) reconnecting [ 1899.079322] LustreError: 57541:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1901.153166] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1903.216258] Lustre: DEBUG MARKER: done [ 1908.193370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.201620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.210533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1908.712506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.721452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.734611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1913.824486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1914.520225] Lustre: server umount lustre-OST0000 complete [ 1918.944716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.973075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.978199] Lustre: Skipped 2 previous similar messages [ 1924.072514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1924.075575] Lustre: Skipped 2 previous similar messages [ 1924.386128] Lustre: server umount lustre-MDT0000 complete [ 1927.737156] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782531991 with bad export cookie 2749568908673191326 [ 1927.743624] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.754987] LustreError: 57527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1928.172148] Lustre: server umount lustre-MDT0001 complete [ 1936.789090] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1939.091722] Key type lgssc unregistered [ 1939.386521] LNet: 61861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.393294] LNetError: 61861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.416562] LNet: Removed LNI 192.168.201.103@tcp [ 1940.033572] Key type .llcrypt unregistered [ 1940.035336] Key type ._llcrypt unregistered [ 1973.210850] Key type ._llcrypt registered [ 1973.212731] Key type .llcrypt registered [ 1973.288084] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1974.144205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.193817] alg: No test for adler32 (adler32-zlib) [ 1975.153707] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 1975.318667] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1976.975207] Key type lgssc registered [ 1977.881727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1980.465919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1980.479874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.748382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1981.770656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1981.937364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.779714] Lustre: Failing over lustre-MDT0000 [ 1983.111924] Lustre: server umount lustre-MDT0000 complete [ 1988.534691] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:47:31 (1782532051) [ 1994.208086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1994.462731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.795323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2004.787392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.854187] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2004.872347] Lustre: 64636:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2005.122725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2008.833653] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2014.712829] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2019.455293] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2025.081967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.200890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2025.354082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.274654] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2031.205436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2035.971744] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2041.500307] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2047.308425] Lustre: server umount lustre-MDT0000 complete [ 2050.493289] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532114 with bad export cookie 14966419392027827762 [ 2050.494181] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.504370] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2050.770374] Lustre: server umount lustre-MDT0001 complete [ 2053.487569] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2056.694146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2062.974042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.679161] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532115/real 1782532115] req@00000000eb0b83f1 x1869120315363712/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2067.701382] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.716647] Lustre: 63564:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532119/real 1782532119] req@0000000067d98aec x1869120315363968/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.735036] Lustre: 63564:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2075.107301] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcfb367243d562715 [ 2075.114619] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2075.404862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.469716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2076.383129] Lustre: 63561:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532124/real 1782532124] req@00000000aa795f53 x1869120315364224/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2076.421932] Lustre: 63561:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2077.475894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2078.260788] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2081.567146] Lustre: 63563:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532129/real 1782532129] req@000000001397d8bc x1869120315364608/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.591596] Lustre: 63563:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2082.797508] LustreError: 68617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.809910] LustreError: 68617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2084.716595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.937753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.738387] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2089.960509] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2092.736881] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2096.501635] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2103.283881] Lustre: server umount lustre-MDT0000 complete [ 2105.316786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2105.327043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.331883] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.375488] Lustre: Skipped 3 previous similar messages [ 2106.127333] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532170 with bad export cookie 14966419392027830037 [ 2106.131616] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.135176] LustreError: 64630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2106.406997] Lustre: server umount lustre-MDT0001 complete [ 2111.180360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2117.172603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.815202] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532174/real 1782532174] req@00000000cdeecfd8 x1869120315387648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.837031] Lustre: 63562:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2126.841934] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.999935] LustreError: 63560:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc920317 x1869120315388800/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.215485] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.223417] LustreError: 66483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2132.283469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.312049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2134.771896] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2136.355348] LustreError: 69145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.364572] LustreError: 69145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2137.376684] Lustre: 63561:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782532185/real 1782532185] req@00000000da35d287 x1869120315388288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782532201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2137.403384] Lustre: 63561:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2140.150443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.931643] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2145.442697] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2147.566460] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.304606] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2156.004984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.011620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.017095] Lustre: Skipped 2 previous similar messages [ 2156.031317] Lustre: Skipped 2 previous similar messages [ 2161.125643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2161.131755] Lustre: Skipped 2 previous similar messages [ 2161.382791] Lustre: server umount lustre-MDT0000 complete [ 2163.956751] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532227 with bad export cookie 14966419392027831416 [ 2163.959267] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.964878] LustreError: 64628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2164.203816] Lustre: server umount lustre-MDT0001 complete [ 2169.318102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2171.571862] Lustre: DEBUG MARKER: files compared the same [ 2183.025223] Lustre: server umount lustre-OST0000 complete [ 2194.206508] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2195.939515] Key type lgssc unregistered [ 2196.146265] LNet: 74362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.156459] LNetError: 74362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2196.166224] LNet: Removed LNI 192.168.201.103@tcp [ 2196.711683] Key type .llcrypt unregistered [ 2196.714084] Key type ._llcrypt unregistered [ 2211.782317] Key type ._llcrypt registered [ 2211.784233] Key type .llcrypt registered [ 2211.856219] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2212.604652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.639804] alg: No test for adler32 (adler32-zlib) [ 2213.632824] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2213.793649] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2215.431913] Key type lgssc registered [ 2216.290303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2221.203972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2221.213985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.514409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.679405] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2231.517166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.784480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2234.817192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2240.117748] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2243.980905] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2248.518124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2248.734943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.390205] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2256.988880] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2267.176584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2277.344391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.350063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.363803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.198091] Lustre: server umount lustre-OST0000 complete [ 2282.470632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.480111] Lustre: Skipped 2 previous similar messages [ 2282.484074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2282.492399] Lustre: Skipped 2 previous similar messages [ 2287.540228] Lustre: server umount lustre-MDT0000 complete [ 2287.590325] LustreError: 75758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.607982] LustreError: 75758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2290.317067] LustreError: 75737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532354 with bad export cookie 6673979075477377842 [ 2290.318848] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.322479] LustreError: 75737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2290.490961] Lustre: server umount lustre-MDT0001 complete [ 2297.437843] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2298.910490] Key type lgssc unregistered [ 2299.083471] LNet: 79311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.087364] LNetError: 79311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.099720] LNet: Removed LNI 192.168.201.103@tcp [ 2299.566147] Key type .llcrypt unregistered [ 2299.568672] Key type ._llcrypt unregistered [ 2314.474200] Key type ._llcrypt registered [ 2314.476311] Key type .llcrypt registered [ 2314.533303] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2315.098471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.131395] alg: No test for adler32 (adler32-zlib) [ 2316.046914] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2316.186596] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2317.839209] Key type lgssc registered [ 2318.406411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2323.431788] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2323.453034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.911478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2344.031316] LustreError: 80694:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 2370.719285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.242789] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2378.753988] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2383.608978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.694559] Lustre: *** cfs_fail_loc=706, val=0*** [ 2383.898256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.936160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2386.122601] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2391.010111] LustreError: 81778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.027728] LustreError: 81778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2391.368058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.542969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.255940] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2398.908934] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2402.490253] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2404.320487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2404.323223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.329809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.884178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2406.888317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.890207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.905463] Lustre: Skipped 1 previous similar message [ 2412.002905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.008681] Lustre: Skipped 1 previous similar message [ 2417.123068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2418.143149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2418.261202] Lustre: server umount lustre-OST0000 complete [ 2422.243655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2422.249842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.250930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2422.250936] Lustre: Skipped 1 previous similar message [ 2422.275608] Lustre: Skipped 1 previous similar message [ 2426.619127] Lustre: server umount lustre-MDT0000 complete [ 2427.360409] LustreError: 81784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.367255] LustreError: 81784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2428.591395] LustreError: 80702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532492 with bad export cookie 16966658066847310302 [ 2428.595943] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2428.601063] LustreError: 80702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2428.790491] Lustre: server umount lustre-MDT0001 complete [ 2434.337806] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2435.780923] Key type lgssc unregistered [ 2435.963624] LNet: 84308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.967245] LNetError: 84308:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.980505] LNet: Removed LNI 192.168.201.103@tcp [ 2436.453197] Key type .llcrypt unregistered [ 2436.454837] Key type ._llcrypt unregistered [ 2451.686551] Key type ._llcrypt registered [ 2451.689904] Key type .llcrypt registered [ 2451.750371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2452.520150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.537000] alg: No test for adler32 (adler32-zlib) [ 2453.481424] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2453.624949] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2455.223166] Key type lgssc registered [ 2455.899354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.854496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.874810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.408544] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2469.410916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.700696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2472.189738] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2474.977594] LustreError: 86390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.987652] LustreError: 86390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2477.465839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2477.647375] LustreError: 86388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.657488] LustreError: 86388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2477.662569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.720222] LustreError: 86393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.729392] LustreError: 86393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2480.949325] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2485.218046] LustreError: 86394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.225975] LustreError: 86394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2485.825347] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.835970] Lustre: MGS: Client 01704c47-d587-4751-9d74-2ce8e2bac95d (at 0@lo) reconnecting [ 2485.855862] LustreError: 86404:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2485.996679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.034720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2488.274100] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2501.609913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.615506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.632756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.638260] Lustre: Skipped 1 previous similar message [ 2504.115709] Lustre: server umount lustre-OST0000 complete [ 2506.724371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.733461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.737286] Lustre: Skipped 2 previous similar messages [ 2506.737849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2511.841229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.844307] Lustre: Skipped 2 previous similar messages [ 2512.629506] Lustre: server umount lustre-MDT0000 complete [ 2515.155421] Lustre: server umount lustre-MDT0001 complete [ 2517.407799] Lustre: server umount MGS complete [ 2524.020033] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:56:27 (1782532587) [ 2541.852470] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2547.847440] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2569.080735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.040361] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2577.628967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2577.784654] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2577.812485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2577.862969] Lustre: lustre-MDT0001: new disk, initializing [ 2577.914904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.934780] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2577.941411] Lustre: 93551:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2577.946486] LustreError: 93551:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2580.807687] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2587.539177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.684695] Lustre: lustre-OST0000: new disk, initializing [ 2587.686779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2587.691440] Lustre: 94318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2587.741788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.760635] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2595.849120] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2596.896188] Lustre: 95015:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000d2009a32 x1869120816582144/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782532676 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2599.341272] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.347042] Lustre: MGS: Client 571fef82-f191-4932-af72-6d9a5d799b2c (at 0@lo) reconnecting [ 2599.369403] Lustre: 92818:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2599.377279] LustreError: 93536:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2599.386341] LustreError: 93536:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2599.394151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2599.453297] Lustre: lustre-MDT0000: new disk, initializing [ 2599.543502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.568184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2600.612094] Lustre: 95387:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000047f9db86 x1869120816587648/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782532680 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2600.672564] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2601.683208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2602.664454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2602.672346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2602.745077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2622.946479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.957130] Lustre: Skipped 1 previous similar message [ 2622.957161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2622.964446] Lustre: Skipped 1 previous similar message [ 2628.074195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.082923] Lustre: Skipped 1 previous similar message [ 2628.405746] Lustre: server umount lustre-OST0000 complete [ 2633.184624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.186214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2636.702166] Lustre: server umount lustre-MDT0000 complete [ 2638.308035] LustreError: 93521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.330218] LustreError: 93521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2653.151138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2653.334810] Lustre: server umount lustre-MDT0001 complete [ 2655.545753] Lustre: server umount MGS complete [ 2661.585090] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:58:44 (1782532724) [ 2671.047198] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2672.635888] Key type lgssc unregistered [ 2672.816740] LNet: 97883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2672.821057] LNetError: 97883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2672.829350] LNet: Removed LNI 192.168.201.103@tcp [ 2673.288177] Key type .llcrypt unregistered [ 2673.289603] Key type ._llcrypt unregistered [ 2682.446743] Key type ._llcrypt registered [ 2682.448726] Key type .llcrypt registered [ 2682.506722] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2682.963584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.061886] alg: No test for adler32 (adler32-zlib) [ 2683.983199] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2684.128879] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2685.743190] Key type lgssc registered [ 2686.301668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2690.417708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2690.426281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.669890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.044660] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2698.883198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.143565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2701.267872] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2705.224279] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2708.249487] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2712.235739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.425093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.438569] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2716.523751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2719.204678] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2729.958197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.959782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2729.964788] Lustre: Skipped 1 previous similar message [ 2729.965257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.969763] Lustre: Skipped 1 previous similar message [ 2733.035468] Lustre: server umount lustre-OST0000 complete [ 2735.072417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.073274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.077959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.081170] Lustre: Skipped 1 previous similar message [ 2740.192778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.198662] Lustre: Skipped 2 previous similar messages [ 2740.842454] Lustre: server umount lustre-MDT0000 complete [ 2743.064363] LustreError: 99064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532807 with bad export cookie 5702817272402970709 [ 2743.069459] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2743.074387] LustreError: 99064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2743.251972] Lustre: server umount lustre-MDT0001 complete [ 2748.848521] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2750.162457] Key type lgssc unregistered [ 2750.345508] LNet: 102613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2750.348990] LNetError: 102613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2750.359544] LNet: Removed LNI 192.168.201.103@tcp [ 2750.816486] Key type .llcrypt unregistered [ 2750.818083] Key type ._llcrypt unregistered [ 2759.652250] Key type ._llcrypt registered [ 2759.654133] Key type .llcrypt registered [ 2759.704350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2760.297992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.321333] alg: No test for adler32 (adler32-zlib) [ 2761.235515] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2761.372657] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2762.976192] Key type lgssc registered [ 2763.632645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.511226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.515947] LustreError: 103673:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2767.525538] LustreError: 103676:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2767.529572] LustreError: 103673:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2767.533908] LustreError: 103673:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2767.539127] LustreError: 103673:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2767.542914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.803316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.882663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.055607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2775.844963] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2778.706438] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lsmod [ 2780.315947] LustreError: 104875:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2780.339221] LustreError: 104882:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2780.342127] LustreError: 104875:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 2780.345736] LustreError: 104875:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2780.350934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2780.352340] LustreError: 104875:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2780.551517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2783.012090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.015262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2783.022085] Lustre: Skipped 2 previous similar messages [ 2783.024809] Lustre: Skipped 1 previous similar message [ 2788.165595] Lustre: server umount lustre-MDT0000 complete [ 2788.322169] LustreError: 103713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.330771] LustreError: 103713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2790.523854] LustreError: 103692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532854 with bad export cookie 7678155094571219032 [ 2790.525081] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2790.535981] LustreError: 103692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2791.263267] LustreError: 104144:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b51ff3e x1869121139525888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2791.376207] Lustre: server umount lustre-MDT0001 complete [ 2803.793795] Lustre: server umount lustre-OST0000 complete [ 2808.169929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.395920] LustreError: 105958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.435378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.321365] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2813.920963] LustreError: 105959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.342498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.540540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2819.353628] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2823.168266] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2826.223602] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2829.860704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.050711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.143108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2833.195840] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2837.375596] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2846.688651] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.691934] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.694785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.696481] Lustre: Skipped 1 previous similar message [ 2846.700278] Lustre: Skipped 2 previous similar messages [ 2847.822150] Lustre: server umount lustre-OST0000 complete [ 2851.809170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.812994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.813265] Lustre: Skipped 1 previous similar message [ 2851.819302] Lustre: Skipped 1 previous similar message [ 2856.930389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.934334] Lustre: Skipped 1 previous similar message [ 2862.048170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.053403] Lustre: Skipped 1 previous similar message [ 2864.095144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2864.176914] Lustre: server umount lustre-MDT0000 complete [ 2866.037332] LustreError: 105941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782532930 with bad export cookie 7678155094571219732 [ 2866.041802] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2866.044840] LustreError: 105941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2866.174609] Lustre: server umount lustre-MDT0001 complete [ 2871.725411] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2873.075144] Key type lgssc unregistered [ 2873.217493] LNet: 109487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.221613] LNetError: 109487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2873.230740] LNet: Removed LNI 192.168.201.103@tcp [ 2873.637713] Key type .llcrypt unregistered [ 2873.638949] Key type ._llcrypt unregistered [ 2886.051781] Key type ._llcrypt registered [ 2886.056281] Key type .llcrypt registered [ 2886.119822] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2886.504847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2886.521345] alg: No test for adler32 (adler32-zlib) [ 2887.456704] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2887.602742] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2889.215157] Key type lgssc registered [ 2889.736474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2893.497551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2893.514246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.796616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.482629] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2902.248976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.431336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.478939] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2908.143322] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2910.847786] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2914.448927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.646653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.762806] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2919.784898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2921.433971] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.008249] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2935.962488] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2938.336712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.341995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.349573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.351563] Lustre: Skipped 1 previous similar message [ 2943.458312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.464712] Lustre: Skipped 1 previous similar message [ 2944.292868] Lustre: server umount lustre-OST0000 complete [ 2948.577555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2948.577764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.582441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.590413] Lustre: Skipped 2 previous similar messages [ 2953.696389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.699082] Lustre: Skipped 2 previous similar messages [ 2958.816873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2958.822782] Lustre: Skipped 1 previous similar message [ 2961.375134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2961.454821] Lustre: server umount lustre-MDT0000 complete [ 2963.592598] LustreError: 113433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533027 with bad export cookie 2197502961149688173 [ 2963.596827] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.601507] LustreError: 113433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2963.846060] Lustre: server umount lustre-MDT0001 complete [ 2969.666184] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2971.043376] Key type lgssc unregistered [ 2971.200634] LNet: 114537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2971.205341] LNetError: 114537:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2972.266872] LNet: Removed LNI 192.168.201.103@tcp [ 2972.783043] Key type .llcrypt unregistered [ 2972.784731] Key type ._llcrypt unregistered [ 2982.837776] Key type ._llcrypt registered [ 2982.839232] Key type .llcrypt registered [ 2982.887607] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2983.388143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2983.411742] alg: No test for adler32 (adler32-zlib) [ 2984.320621] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 2984.437084] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2986.047197] Key type lgssc registered [ 2986.696248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.741607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.752344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.957563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.132835] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2998.522215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.742756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3001.060133] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3004.864514] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.651929] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3011.160038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.316372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.049680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3016.434471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3017.495772] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3028.964481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.964694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.965770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.970568] Lustre: Skipped 1 previous similar message [ 3029.320498] Lustre: server umount lustre-OST0000 complete [ 3034.080961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.091564] Lustre: Skipped 1 previous similar message [ 3034.093869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.105502] Lustre: Skipped 2 previous similar messages [ 3037.636341] Lustre: server umount lustre-MDT0000 complete [ 3039.201119] LustreError: 116794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.210218] LustreError: 116794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3039.808971] LustreError: 115719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533103 with bad export cookie 2069349105276333633 [ 3039.810141] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3039.813248] LustreError: 115719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3039.983709] Lustre: server umount lustre-MDT0001 complete [ 3045.789584] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3047.269893] Key type lgssc unregistered [ 3047.418978] LNet: 119269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3047.423929] LNetError: 119269:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3047.435478] LNet: Removed LNI 192.168.201.103@tcp [ 3047.871310] Key type .llcrypt unregistered [ 3047.872808] Key type ._llcrypt unregistered [ 3060.035216] Key type ._llcrypt registered [ 3060.036405] Key type .llcrypt registered [ 3060.085209] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3060.791223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3060.810882] alg: No test for adler32 (adler32-zlib) [ 3061.705211] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3061.822228] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3063.431115] Key type lgssc registered [ 3063.933584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3067.769496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3067.787913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.022749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.152021] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3074.886569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.047109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3076.908212] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3080.389623] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3083.526979] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3086.888320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.048909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.650595] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3093.044296] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.246717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3102.442773] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3104.379535] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3104.382811] Lustre: Skipped 2 previous similar messages [ 3106.442816] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3106.445716] Lustre: Skipped 2 previous similar messages [ 3112.450338] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3112.459396] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3114.953044] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3114.956850] Lustre: Skipped 2 previous similar messages [ 3117.891072] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp [ 3123.039297] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.3/24@tcp in log lustre-MDT0000 [ 3123.045052] Lustre: Skipped 2 previous similar messages [ 3126.219234] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3126.221488] Lustre: Skipped 1 previous similar message [ 3133.489834] Lustre: server umount lustre-OST0000 complete [ 3136.481023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3136.484067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.489554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.491552] Lustre: Skipped 1 previous similar message [ 3141.601078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3141.603133] Lustre: Skipped 1 previous similar message [ 3145.733656] Lustre: server umount lustre-MDT0000 complete [ 3146.720345] LustreError: 120663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.733432] LustreError: 120663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3147.533290] LustreError: 120642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533211 with bad export cookie 13739740919808090815 [ 3147.536771] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.537776] LustreError: 120642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3158.039510] Lustre: server umount lustre-MDT0001 complete [ 3163.063480] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3164.374764] Key type lgssc unregistered [ 3164.556587] LNet: 125591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.560955] LNetError: 125591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.570397] LNet: Removed LNI 192.168.201.103@tcp [ 3164.951916] Key type .llcrypt unregistered [ 3164.954912] Key type ._llcrypt unregistered [ 3178.309289] Key type ._llcrypt registered [ 3178.310908] Key type .llcrypt registered [ 3178.360784] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3178.707509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3178.748094] alg: No test for adler32 (adler32-zlib) [ 3179.684613] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3179.827193] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3181.431168] Key type lgssc registered [ 3182.032178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3185.680643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3185.696786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.893973] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3186.905478] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.3/24@tcp [ 3186.912375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.775108] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3192.546295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.701801] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3192.711162] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3192.731085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3194.534680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3197.796974] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3200.345072] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3203.612178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3203.745245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.330066] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3209.362476] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3211.949818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3227.619352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.619997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.621913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.625621] LustreError: Skipped 1 previous similar message [ 3227.630543] Lustre: Skipped 1 previous similar message [ 3229.028252] Lustre: server umount lustre-OST0000 complete [ 3232.736900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.738670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.744921] Lustre: Skipped 1 previous similar message [ 3232.751667] Lustre: Skipped 2 previous similar messages [ 3237.041837] Lustre: server umount lustre-MDT0000 complete [ 3237.857459] LustreError: 127130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.869241] LustreError: 127130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3238.676261] LustreError: 127111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533302 with bad export cookie 66642213125431716 [ 3238.679130] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.680400] LustreError: 127111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3238.806733] Lustre: server umount lustre-MDT0001 complete [ 3243.322863] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3244.519199] Key type lgssc unregistered [ 3244.661395] LNet: 130734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.667334] LNetError: 130734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.683281] LNet: Removed LNI 192.168.201.103@tcp [ 3245.051260] Key type .llcrypt unregistered [ 3245.052595] Key type ._llcrypt unregistered [ 3254.901096] Key type ._llcrypt registered [ 3254.902484] Key type .llcrypt registered [ 3254.943932] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3261.916143] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3262.305349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3262.340214] alg: No test for adler32 (adler32-zlib) [ 3263.253385] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3263.377940] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3264.975135] Key type lgssc registered [ 3265.466133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3279.151139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3279.157580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.255243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3280.268499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3280.307327] Lustre: lustre-MDT0000: new disk, initializing [ 3280.332149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.338660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3281.787750] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3286.191528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.242323] Lustre: 134106:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3286.273482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3286.276122] Lustre: Skipped 1 previous similar message [ 3286.314268] Lustre: lustre-MDT0001: new disk, initializing [ 3286.354654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.368850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3286.372552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3287.962564] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3292.359068] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3294.781342] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3297.572677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.709905] Lustre: lustre-OST0000: new disk, initializing [ 3297.711911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3297.715279] Lustre: 136076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3297.751339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.153064] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3304.693218] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3306.521348] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3307.009050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3307.013702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3307.058901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3307.649434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3309.369377] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3309.458858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3312.098703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.098779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3312.099880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3312.103213] Lustre: Skipped 1 previous similar message [ 3312.106142] LustreError: Skipped 1 previous similar message [ 3316.310470] Lustre: server umount lustre-OST0000 complete [ 3322.336289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3322.336794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.337573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.337577] Lustre: Skipped 1 previous similar message [ 3322.348061] Lustre: Skipped 1 previous similar message [ 3323.812534] Lustre: server umount lustre-MDT0000 complete [ 3325.350562] LustreError: 134096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533389 with bad export cookie 2578283743593912000 [ 3325.353824] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3325.354992] LustreError: 134096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3325.481430] Lustre: server umount lustre-MDT0001 complete [ 3328.860363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.103221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.801400] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3334.112621] LustreError: 138297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.118503] LustreError: 138297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3334.125860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.283995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3335.826524] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3338.760787] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3341.370562] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3344.597096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.803712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.487471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3350.547989] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3352.273569] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3354.465467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3356.299932] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3359.470182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3364.304742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3364.433184] Lustre: lustre-OST0001: new disk, initializing [ 3364.435877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3364.441043] Lustre: 141700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3364.495430] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.792892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3370.022736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3370.027185] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3370.065304] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3372.013198] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3372.105754] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3374.201229] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3374.286958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3376.814137] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97dd05ed0800.ost_server_uuid 50 [ 3377.439251] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97dd05ed0800.ost_server_uuid in FULL state after 0 sec [ 3385.312535] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3385.313676] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.317055] LustreError: Skipped 1 previous similar message [ 3385.318053] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3385.326059] Lustre: Skipped 1 previous similar message [ 3385.327954] Lustre: Skipped 1 previous similar message [ 3386.626780] Lustre: server umount lustre-OST0001 complete [ 3389.920447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.921361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.926217] LustreError: Skipped 1 previous similar message [ 3389.926960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.931339] Lustre: Skipped 1 previous similar message [ 3390.435045] LustreError: 140116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.440349] LustreError: 140116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3394.279822] Lustre: server umount lustre-OST0000 complete [ 3396.065617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3396.069612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.076222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3396.079714] Lustre: Skipped 3 previous similar messages [ 3405.794113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.798814] Lustre: Skipped 3 previous similar messages [ 3410.399405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3410.468779] Lustre: server umount lustre-MDT0000 complete [ 3410.912273] LustreError: 138297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.917451] LustreError: 138297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3411.936783] LustreError: 138279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533475 with bad export cookie 2578283743593912868 [ 3411.939330] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3411.940866] LustreError: 138279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3412.079703] Lustre: server umount lustre-MDT0001 complete [ 3419.781959] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3421.058665] Key type lgssc unregistered [ 3421.215522] LNet: 144700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.218988] LNetError: 144700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.226333] LNet: Removed LNI 192.168.201.103@tcp [ 3421.573126] Key type .llcrypt unregistered [ 3421.574622] Key type ._llcrypt unregistered [ 3438.909233] Key type ._llcrypt registered [ 3438.910562] Key type .llcrypt registered [ 3438.952230] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3446.261115] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3446.728344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3446.738492] alg: No test for adler32 (adler32-zlib) [ 3447.607780] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3447.705468] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3449.311365] Key type lgssc registered [ 3449.850286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3465.098187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3465.115287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.249316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3466.267470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3466.306142] Lustre: lustre-MDT0000: new disk, initializing [ 3466.333981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3466.343158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3468.055732] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3473.923501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.973937] Lustre: 148826:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3474.009802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3474.011758] Lustre: Skipped 1 previous similar message [ 3474.054239] Lustre: lustre-MDT0001: new disk, initializing [ 3474.094414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3474.109684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3474.118587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3475.937496] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3480.119326] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3482.202131] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3484.861905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.974146] Lustre: lustre-OST0000: new disk, initializing [ 3484.976400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3484.979873] Lustre: 150794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3485.011234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.371234] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3491.704667] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3492.346476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3492.350169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3492.387974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3499.178697] Lustre: Failing over lustre-OST0000 [ 3499.264065] Lustre: server umount lustre-OST0000 complete [ 3513.202228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.398024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.415102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.422081] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3513.606761] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3515.693459] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3517.234290] Lustre: Failing over lustre-MDT0000 [ 3517.347392] Lustre: server umount lustre-MDT0000 complete [ 3518.431101] Lustre: 147053:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782533566/real 1782533566] req@000000000235caa9 x1869121858859520/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782533582 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3518.432386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.444323] Lustre: Skipped 2 previous similar messages [ 3523.551209] Lustre: 147056:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782533571/real 1782533571] req@00000000a6f8a1d5 x1869121858861184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782533587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3523.552943] LustreError: 148833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.560933] Lustre: 147056:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3523.567844] LustreError: 148833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3528.671392] Lustre: 147055:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782533576/real 1782533576] req@0000000065df5cef x1869121858862336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782533592 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3528.672958] LustreError: 148836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3528.691700] LustreError: 148836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3530.960512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.029525] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.033909] Lustre: 148828:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.103@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3531.123655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3531.143604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.168381] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3532.724915] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3534.076277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3536.360634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3536.369281] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3536.397771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3541.476272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3541.477347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.481446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.486431] Lustre: Skipped 2 previous similar messages [ 3543.703496] Lustre: server umount lustre-OST0000 complete [ 3546.592525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.592829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3546.598702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.601066] LustreError: Skipped 1 previous similar message [ 3546.603408] Lustre: Skipped 2 previous similar messages [ 3551.712077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.714249] Lustre: Skipped 1 previous similar message [ 3556.832098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.834094] Lustre: Skipped 1 previous similar message [ 3559.391113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3559.463477] Lustre: server umount lustre-MDT0000 complete [ 3561.146606] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533625 with bad export cookie 10055618020099139966 [ 3561.147921] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.152629] LustreError: 148816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3561.300633] Lustre: server umount lustre-MDT0001 complete [ 3565.693406] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3566.951084] Key type lgssc unregistered [ 3567.123915] LNet: 154443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3567.128451] LNetError: 154443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3567.137518] LNet: Removed LNI 192.168.201.103@tcp [ 3567.531675] Key type .llcrypt unregistered [ 3567.533073] Key type ._llcrypt unregistered [ 3589.638256] Key type ._llcrypt registered [ 3589.639457] Key type .llcrypt registered [ 3589.682939] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3590.137561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3590.155860] alg: No test for adler32 (adler32-zlib) [ 3591.015308] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3591.117093] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3592.703159] Key type lgssc registered [ 3593.159225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3596.342283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3596.350043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3597.551964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.377557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3603.619850] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3605.659901] LustreError: 155991:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3605.665177] LustreError: 155991:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3605.668883] LustreError: 155991:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 3605.794071] Lustre: server umount lustre-MDT0000 complete [ 3611.410297] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3614.979173] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3628.446033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.546173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3628.559424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3628.597780] Lustre: lustre-MDT0000: new disk, initializing [ 3628.623902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3628.632488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3630.087231] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3634.812622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.866118] Lustre: 159288:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3634.898601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3634.901853] Lustre: Skipped 1 previous similar message [ 3634.943997] Lustre: lustre-MDT0001: new disk, initializing [ 3634.990148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3635.005386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3635.011484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3636.682667] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3640.870575] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3643.060580] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3645.623740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.747267] Lustre: lustre-OST0000: new disk, initializing [ 3645.749525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3645.755700] Lustre: 161254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3645.802223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.242509] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3652.365987] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3653.629116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.633245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3653.670584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3659.756398] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3660.347589] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:15:24 (1782533724) [ 3740.639477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.639704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3740.643219] Lustre: Skipped 2 previous similar messages [ 3740.647018] Lustre: Skipped 2 previous similar messages [ 3744.311223] Lustre: server umount lustre-MDT0000 complete [ 3745.627801] LustreError: 159279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533809 with bad export cookie 2849834958339432925 [ 3745.628809] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3745.632320] LustreError: 159279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3745.759488] LustreError: 159293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.759540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.767479] LustreError: 159293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3745.772151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3750.879743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3750.879780] LustreError: 160094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.778924] Lustre: server umount lustre-MDT0001 complete [ 3759.185129] Lustre: server umount lustre-OST0000 complete [ 3762.364088] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3765.909854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3778.368025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.455831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3778.465581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3778.501773] Lustre: lustre-MDT0000: new disk, initializing [ 3778.527963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3778.535023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3779.900638] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3784.128725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3784.167767] Lustre: 165885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3784.190665] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3784.192224] Lustre: Skipped 1 previous similar message [ 3784.227266] Lustre: lustre-MDT0001: new disk, initializing [ 3784.269524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3784.275204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3785.710661] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3789.681554] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3791.571136] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3794.071224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.171797] Lustre: lustre-OST0000: new disk, initializing [ 3794.173739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3794.177393] Lustre: 167853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3794.222101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.224634] Lustre: Skipped 1 previous similar message [ 3796.425090] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3799.529832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3799.534447] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3799.548491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3800.463957] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3802.026589] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3802.107824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3803.714507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3803.803381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3804.705749] Lustre: server umount lustre-OST0000 complete [ 3809.761322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3809.761396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.764767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3809.769683] Lustre: Skipped 1 previous similar message [ 3814.880278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3814.883314] Lustre: Skipped 2 previous similar messages [ 3816.967505] Lustre: server umount lustre-MDT0000 complete [ 3818.484589] LustreError: 165876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533882 with bad export cookie 2849834958339591741 [ 3818.486217] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3818.489187] LustreError: 165876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3828.761734] Lustre: server umount lustre-MDT0001 complete [ 3831.898052] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:18:15 (1782533895) [ 3837.790507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3841.524707] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3855.094335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.184624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3855.197399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3855.234892] Lustre: lustre-MDT0000: new disk, initializing [ 3855.264316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.271307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3856.722940] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3861.018164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.067226] Lustre: 172603:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3861.165870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3862.650657] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3866.582932] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3868.475378] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3870.885303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3871.005640] Lustre: 174576:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3873.048953] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3876.937947] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3877.879070] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3877.913888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3888.097106] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.098848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.101029] LustreError: Skipped 1 previous similar message [ 3888.105953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.109188] Lustre: Skipped 1 previous similar message [ 3890.451746] Lustre: server umount lustre-OST0000 complete [ 3892.192410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3892.195865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3892.200901] Lustre: Skipped 1 previous similar message [ 3898.076583] Lustre: server umount lustre-MDT0000 complete [ 3898.336066] LustreError: 175256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.341318] LustreError: 175256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3899.494485] LustreError: 173401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782533963 with bad export cookie 2849834958339592609 [ 3899.497390] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3899.499345] LustreError: 173401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3899.626095] Lustre: server umount lustre-MDT0001 complete [ 3902.860790] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:19:26 (1782533966) [ 3908.794164] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3911.973506] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3925.077630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.167543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3925.181042] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3925.183479] Lustre: Skipped 3 previous similar messages [ 3925.218914] Lustre: lustre-MDT0000: new disk, initializing [ 3925.220631] Lustre: Skipped 2 previous similar messages [ 3925.242778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.246048] Lustre: Skipped 2 previous similar messages [ 3925.252184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3925.255344] Lustre: Skipped 2 previous similar messages [ 3926.783962] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3931.177709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3931.225823] Lustre: 178891:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.330508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3932.827571] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3936.825041] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3938.740575] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3941.137367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3941.238636] Lustre: 180869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3942.686255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3943.378985] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3947.320920] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3953.120335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3953.121409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.123510] LustreError: Skipped 1 previous similar message [ 3953.127692] Lustre: Skipped 1 previous similar message [ 3953.128711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3953.132603] Lustre: Skipped 4 previous similar messages [ 3955.412680] Lustre: server umount lustre-OST0000 complete [ 3957.215902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3962.845236] Lustre: server umount lustre-MDT0000 complete [ 3963.360993] LustreError: 178902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.368100] LustreError: 178902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3964.248155] LustreError: 178883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534028 with bad export cookie 2849834958339593694 [ 3964.251090] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.254601] LustreError: 178883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3964.378518] Lustre: server umount lustre-MDT0001 complete [ 3968.605609] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3969.688945] Key type lgssc unregistered [ 3969.819395] LNet: 182723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3969.822153] LNetError: 182723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3969.831306] LNet: Removed LNI 192.168.201.103@tcp [ 3970.148127] Key type .llcrypt unregistered [ 3970.149690] Key type ._llcrypt unregistered [ 3979.949501] Key type ._llcrypt registered [ 3979.951142] Key type .llcrypt registered [ 3979.993026] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3980.307498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3980.315530] alg: No test for adler32 (adler32-zlib) [ 3981.182731] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 3981.280279] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3982.879136] Key type lgssc registered [ 3983.290851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3986.238558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3986.246868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.419130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3988.913903] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3992.256013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.394511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.954336] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3996.781844] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3998.840125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4001.523834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4001.658887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.883647] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4006.598868] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4012.967509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4028.385720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4028.387469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.388546] LustreError: Skipped 1 previous similar message [ 4028.388980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4028.391981] Lustre: Skipped 1 previous similar message [ 4031.254467] Lustre: server umount lustre-OST0000 complete [ 4033.504664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4033.504972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4033.512859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4033.515688] Lustre: Skipped 2 previous similar messages [ 4038.624293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4038.626882] Lustre: Skipped 1 previous similar message [ 4038.751734] Lustre: server umount lustre-MDT0000 complete [ 4040.091382] LustreError: 186614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534104 with bad export cookie 12077605241101884358 [ 4040.094470] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.095665] LustreError: 186614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4040.219653] Lustre: server umount lustre-MDT0001 complete [ 4044.084046] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4045.163171] Key type lgssc unregistered [ 4045.310682] LNet: 187665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4045.313520] LNetError: 187665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4045.323257] LNet: Removed LNI 192.168.201.103@tcp [ 4045.662880] Key type .llcrypt unregistered [ 4045.663961] Key type ._llcrypt unregistered [ 4055.726426] Key type ._llcrypt registered [ 4055.727887] Key type .llcrypt registered [ 4055.770529] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4056.100521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4056.119055] alg: No test for adler32 (adler32-zlib) [ 4056.971039] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 4057.063451] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4058.649671] Key type lgssc registered [ 4059.061055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4061.998081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4062.006163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.167093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.687362] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4067.763835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.889880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4069.282689] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4071.861331] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4073.854108] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4076.365069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4076.492806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.555048] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4081.141781] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4088.806227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4097.796337] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 4098.015986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.020072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.041182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.041994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4099.046167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4099.047836] Lustre: Skipped 2 previous similar messages [ 4102.909644] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 4108.027976] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 4108.030480] Lustre: Skipped 2 previous similar messages [ 4109.279123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.331417] Lustre: server umount lustre-OST0000 complete [ 4111.857382] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97dd20271000.ost_server_uuid 50 [ 4112.418726] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff97dd20271000.ost_server_uuid in DISCONN state after 0 sec [ 4113.044958] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4123.148442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.149138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.153139] Lustre: Skipped 1 previous similar message [ 4123.155333] Lustre: Skipped 2 previous similar messages [ 4129.306990] Lustre: server umount lustre-MDT0000 complete [ 4130.701154] LustreError: 189050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534194 with bad export cookie 15831468562869012113 [ 4130.704335] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.704744] LustreError: 189050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4130.820863] Lustre: server umount lustre-MDT0001 complete [ 4133.849662] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:23:17 (1782534197) [ 4136.163415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.320117] LustreError: 193031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.326338] LustreError: 193031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4136.343080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.760109] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4140.845555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.981411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4142.380085] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4144.899529] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4146.767130] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4149.120904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4149.235984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.236299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4153.861825] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4156.178106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.254140] Lustre: lustre-OST0001: new disk, initializing [ 4156.255981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4156.258453] Lustre: 195868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4158.274845] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4161.446627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4161.533854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4161.537591] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4161.568343] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4162.066343] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4165.599820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4165.602917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.607865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4165.610695] Lustre: Skipped 2 previous similar messages [ 4169.082910] Lustre: server umount lustre-OST0000 complete [ 4171.156181] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4171.237406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4171.746231] LustreError: 195028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.751423] LustreError: 195028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4172.637508] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4172.710682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4176.866066] LustreError: 194848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.872513] LustreError: 194848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4181.988438] LustreError: 194848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.996320] LustreError: 194848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4187.106654] LustreError: 194850:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.107102] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.107943] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.107950] Lustre: Skipped 1 previous similar message [ 4187.108720] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4187.108724] Lustre: Skipped 3 previous similar messages [ 4187.111989] LustreError: 194850:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4187.113933] LustreError: Skipped 1 previous similar message [ 4188.889329] Lustre: server umount lustre-OST0001 complete [ 4192.224107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4196.317554] Lustre: server umount lustre-MDT0000 complete [ 4197.344461] LustreError: 193056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.351197] LustreError: 193056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4197.628250] LustreError: 193012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534261 with bad export cookie 15831468562869013170 [ 4197.629546] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4197.632294] LustreError: 193012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4197.758626] Lustre: server umount lustre-MDT0001 complete [ 4206.045100] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 00:24:29 (1782534269) [ 4208.315697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.410256] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4208.428847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4208.499392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.501841] Lustre: Skipped 1 previous similar message [ 4209.887470] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4212.965601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.001338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4213.016245] Lustre: 199281:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4214.609051] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4217.205555] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4219.154129] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4221.511805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4221.580628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4223.828059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4223.907352] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4226.414433] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4228.951280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.986785] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4230.053201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4231.205709] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4233.827112] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4235.235554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.237070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4235.241105] Lustre: Skipped 4 previous similar messages [ 4235.243092] Lustre: Skipped 3 previous similar messages [ 4249.055105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4249.082468] Lustre: server umount lustre-OST0000 complete [ 4249.341604] LustreError: 201131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.834181] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4260.834256] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4260.836747] LustreError: Skipped 1 previous similar message [ 4265.940550] Lustre: server umount lustre-OST0001 complete [ 4273.447241] Lustre: server umount lustre-MDT0000 complete [ 4274.854187] LustreError: 201160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534338 with bad export cookie 15831468562869014556 [ 4274.857445] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4274.858180] LustreError: 201160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4274.978017] Lustre: server umount lustre-MDT0001 complete [ 4283.361874] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 00:25:47 (1782534347) [ 4288.858074] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4292.370735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4305.377986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4305.463677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4305.477422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4305.515459] Lustre: lustre-MDT0000: new disk, initializing [ 4305.541501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.544592] Lustre: Skipped 3 previous similar messages [ 4305.552488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4307.041107] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4311.664145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.716272] Lustre: 207345:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4311.745330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4311.747584] Lustre: Skipped 1 previous similar message [ 4311.785742] Lustre: lustre-MDT0001: new disk, initializing [ 4311.838209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4311.841371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4313.368600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4317.279568] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4319.291091] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4321.832700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4321.942378] Lustre: lustre-OST0000: new disk, initializing [ 4321.944687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4321.947493] Lustre: 209310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4324.154281] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4328.150123] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4329.777591] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4330.488072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4330.491617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4330.529387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4330.892431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4332.419888] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4332.500654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4335.584995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4335.588511] LustreError: Skipped 1 previous similar message [ 4335.590156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.594373] Lustre: Skipped 3 previous similar messages [ 4335.596592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.602857] Lustre: Skipped 13 previous similar messages [ 4339.287363] Lustre: server umount lustre-OST0000 complete [ 4346.839573] Lustre: server umount lustre-MDT0000 complete [ 4348.088357] LustreError: 207336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534412 with bad export cookie 15831468562869016327 [ 4348.089308] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4348.093983] LustreError: 207336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4348.212429] Lustre: server umount lustre-MDT0001 complete [ 4351.084350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.227535] LustreError: 211531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.233562] LustreError: 211531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4352.574326] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4355.648237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4357.184830] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4359.773415] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4361.774426] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4364.322037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4366.546488] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4369.139313] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4370.690476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4377.936311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4379.496610] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4379.585303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4381.153108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4381.156738] LustreError: Skipped 1 previous similar message [ 4386.459728] Lustre: server umount lustre-OST0000 complete [ 4388.676557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4390.182586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4413.946764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.059061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.061762] Lustre: Skipped 5 previous similar messages [ 4416.200354] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4418.683052] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4419.240910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4422.354408] Lustre: server umount lustre-OST0000 complete [ 4425.750169] Lustre: server umount lustre-MDT0000 complete [ 4427.117594] LustreError: 211512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534491 with bad export cookie 15831468562869017195 [ 4427.121311] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4427.121887] LustreError: 211512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4427.243488] Lustre: server umount lustre-MDT0001 complete [ 4430.247736] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 00:28:13 (1782534493) [ 4432.526114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.669185] LustreError: 217534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.992996] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4436.748852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4438.137403] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4440.636692] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4442.466217] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4444.745290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4446.759655] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4449.289440] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4450.955125] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4458.164674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4459.543464] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4459.618742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4461.991084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.060432] Lustre: lustre-OST0001: new disk, initializing [ 4462.061785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4462.063804] Lustre: 220952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4464.067984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4467.825321] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4469.266594] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4469.761547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4469.765393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4469.795354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4470.361206] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4471.859053] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4471.946802] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4474.853618] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4474.854393] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4474.858142] Lustre: Skipped 5 previous similar messages [ 4474.860731] LustreError: Skipped 2 previous similar messages [ 4474.861479] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4474.866787] Lustre: Skipped 8 previous similar messages [ 4478.720264] Lustre: server umount lustre-OST0001 complete [ 4480.912823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4480.994758] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4482.659669] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4482.735676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4506.494040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4508.626764] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4510.996204] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4511.498600] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4511.718771] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4511.722240] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4511.722259] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4514.232263] Lustre: server umount lustre-OST0001 complete [ 4532.436473] Lustre: server umount lustre-OST0000 complete [ 4539.932644] Lustre: server umount lustre-MDT0000 complete [ 4541.192060] LustreError: 217514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534605 with bad export cookie 15831468562869018126 [ 4541.194955] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.195694] LustreError: 217514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4547.383888] Lustre: server umount lustre-MDT0001 complete [ 4555.294157] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 00:30:18 (1782534618) [ 4557.613663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.702676] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4557.711517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4557.781835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.784925] Lustre: Skipped 5 previous similar messages [ 4559.194319] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4562.198309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.231947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4562.243094] Lustre: 226190:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4563.780683] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4566.395423] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4568.392943] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4570.829321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.891182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4573.122484] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4575.766054] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4579.968040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4579.994730] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4582.382867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4585.310956] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4585.953760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4587.452721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4587.530743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4589.039826] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4589.118694] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4591.102142] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97dd200ab800.ost_server_uuid 50 [ 4591.654754] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97dd200ab800.ost_server_uuid in FULL state after 0 sec [ 4592.015576] Lustre: Permanently deactivating lustre-OST0001 [ 4592.020703] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4592.841895] Lustre: Permanently reactivating lustre-OST0001 [ 4599.937895] Lustre: server umount lustre-OST0001 complete [ 4604.385888] LustreError: 228032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.387520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.390900] LustreError: 228032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 4604.394732] Lustre: Skipped 9 previous similar messages [ 4607.376117] Lustre: server umount lustre-OST0000 complete [ 4614.752929] Lustre: server umount lustre-MDT0000 complete [ 4616.138259] LustreError: 226182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534680 with bad export cookie 15831468562869019400 [ 4616.141850] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4616.143202] LustreError: 226182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4616.269565] Lustre: server umount lustre-MDT0001 complete [ 4625.269727] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 00:31:28 (1782534688) [ 4627.870796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.985234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4627.997619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4628.001366] Lustre: Skipped 5 previous similar messages [ 4629.441792] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4632.430391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.463548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4632.472568] Lustre: 232871:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4633.912014] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4636.386731] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.294286] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4640.584646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.662566] Lustre: Permanently deactivating lustre-OST0000 [ 4640.667020] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4641.184360] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4642.642626] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4645.093562] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4647.443604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4649.470667] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4649.620948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4652.108442] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4652.755431] Lustre: Permanently reactivating lustre-OST0000 [ 4657.654129] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4657.656354] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4657.659415] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4657.661678] Lustre: Skipped 1 previous similar message [ 4657.675051] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4657.683418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4664.803692] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4664.806885] LustreError: Skipped 5 previous similar messages [ 4668.099563] Lustre: server umount lustre-OST0001 complete [ 4675.537962] Lustre: server umount lustre-OST0000 complete [ 4683.159670] Lustre: server umount lustre-MDT0000 complete [ 4684.416036] LustreError: 232861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534748 with bad export cookie 15831468562869021038 [ 4684.419290] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.420144] LustreError: 232861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4684.533612] Lustre: server umount lustre-MDT0001 complete [ 4688.159671] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4689.201934] Key type lgssc unregistered [ 4689.329396] LNet: 237722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4689.332133] LNetError: 237722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4689.340339] LNet: Removed LNI 192.168.201.103@tcp [ 4689.662662] Key type .llcrypt unregistered [ 4689.663532] Key type ._llcrypt unregistered [ 4699.074204] Key type ._llcrypt registered [ 4699.075366] Key type .llcrypt registered [ 4699.117855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4699.569961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4699.579775] alg: No test for adler32 (adler32-zlib) [ 4700.442345] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 4700.538780] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4702.127146] Key type lgssc registered [ 4702.518070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4706.261558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4706.269547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.441692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4708.967874] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4712.391400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.439573] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4712.453091] Lustre: 239154:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4712.459294] Lustre: Permanently deactivating lustre-MDT0001 [ 4712.466054] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4712.595267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.096865] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4716.846254] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.036890] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4717.039678] LustreError: 239195:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4717.044131] LustreError: 239195:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4717.047628] LustreError: 239195:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 4718.835124] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4721.144211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4721.258782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.274804] LustreError: 241005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.277653] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4722.281487] LustreError: 241005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4723.252190] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4725.820988] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4727.777977] LustreError: 241007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.290429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4728.372519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4730.390946] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4732.457365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 4732.457988] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 4732.501342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 4732.889466] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4734.485496] Lustre: Permanently reactivating lustre-MDT0001 [ 4734.491209] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 4734.493202] Lustre: Skipped 2 previous similar messages [ 4741.377395] Lustre: lustre-MDT0001: Client 6a58dff2-f309-4da0-b6e3-7150528850ad (at 192.168.201.3@tcp) reconnecting [ 4743.397284] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4743.930567] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4745.545799] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 4745.619569] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 4746.904570] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 4746.973442] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 4747.756342] Lustre: Permanently deactivating lustre-MDT0001 [ 4747.760960] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 4747.763962] Lustre: Skipped 2 previous similar messages [ 4749.051936] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4752.866543] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.867642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4752.871457] Lustre: Skipped 2 previous similar messages [ 4752.874686] Lustre: Skipped 3 previous similar messages [ 4756.377458] Lustre: server umount lustre-MDT0000 complete [ 4757.609151] LustreError: 242707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534821 with bad export cookie 4726336092137485499 [ 4757.612563] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.613184] LustreError: 242707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4757.983764] LustreError: 239159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4757.988419] LustreError: 239159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4759.107736] Lustre: server umount lustre-MDT0001 complete [ 4766.458453] Lustre: server umount lustre-OST0000 complete [ 4770.017599] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:33:53 (1782534833) [ 4773.221348] Lustre: server umount lustre-OST0001 complete [ 4775.113395] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4778.041729] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4789.378121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.470798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.476333] Lustre: Skipped 2 previous similar messages [ 4789.484138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4789.519330] Lustre: lustre-MDT0000: new disk, initializing [ 4789.539651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.546787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4790.878659] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4794.942223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.979970] Lustre: 247405:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4795.002255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4795.003920] Lustre: Skipped 1 previous similar message [ 4795.035730] Lustre: lustre-MDT0001: new disk, initializing [ 4795.080903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4795.084743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4796.409581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4800.101140] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4801.823887] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4804.006116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.098530] Lustre: lustre-OST0000: new disk, initializing [ 4804.100700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4804.103216] Lustre: 249372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4804.130331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.132843] Lustre: Skipped 1 previous similar message [ 4805.931022] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4806.644267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4806.647442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4806.674777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4809.569782] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.230184] LustreError: 250091:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4814.426492] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.486395] Lustre: lustre-OST0001: new disk, initializing [ 4814.488184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4814.490996] Lustre: 250491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4816.354891] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4819.949587] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4820.319061] LustreError: 250091:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4820.328243] LustreError: 247442:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4820.477055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4820.479913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4820.511100] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4820.520802] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4830.415116] LustreError: 247442:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4832.002538] Lustre: lustre-OST0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 4835.810202] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4835.812298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4835.813669] Lustre: Skipped 1 previous similar message [ 4835.816364] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4835.818847] Lustre: Skipped 1 previous similar message [ 4837.116600] LustreError: 251257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.117375] Lustre: server umount lustre-OST0001 complete [ 4846.051024] LustreError: 249357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.056195] LustreError: 249357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4851.170457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.175121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4851.177464] Lustre: Skipped 1 previous similar message [ 4855.056168] Lustre: server umount lustre-OST0000 complete [ 4856.799964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4856.802133] LustreError: Skipped 1 previous similar message [ 4856.803284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.806355] Lustre: Skipped 1 previous similar message [ 4856.807860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.408653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4862.550774] Lustre: server umount lustre-MDT0000 complete [ 4863.772125] LustreError: 247400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782534927 with bad export cookie 4726336092137488306 [ 4863.774396] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4863.776055] LustreError: 247400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4878.303095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4878.394252] Lustre: server umount lustre-MDT0001 complete [ 4881.934262] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4882.980281] Key type lgssc unregistered [ 4883.114407] LNet: 252520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.117767] LNetError: 252520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4883.126302] LNet: Removed LNI 192.168.201.103@tcp [ 4883.424266] Key type .llcrypt unregistered [ 4883.425208] Key type ._llcrypt unregistered [ 4897.034758] Key type ._llcrypt registered [ 4897.035574] Key type .llcrypt registered [ 4897.072070] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4897.373085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4897.411218] alg: No test for adler32 (adler32-zlib) [ 4898.250499] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 4898.332533] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4899.911128] Key type lgssc registered [ 4900.236475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4902.661642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4902.668303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.750137] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4903.759342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4903.810343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4905.047493] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4907.627340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.653437] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4907.662600] Lustre: 254705:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4907.742076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4908.981041] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4911.227950] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4912.958562] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4915.193454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4915.257829] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4915.312572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4916.618144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4917.188900] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4919.519881] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4962.785081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4962.785246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.787271] LustreError: Skipped 1 previous similar message [ 4962.791892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4962.794009] Lustre: Skipped 1 previous similar message [ 4964.247852] Lustre: server umount lustre-OST0000 complete [ 4965.653040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4969.685026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4969.794203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.608305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4973.884552] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4982.051662] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.054752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.054915] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4982.059314] Lustre: Skipped 1 previous similar message [ 5007.841573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5007.842447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.843424] LustreError: Skipped 1 previous similar message [ 5007.843931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5007.847548] Lustre: Skipped 2 previous similar messages [ 5012.818352] Lustre: server umount lustre-OST0000 complete [ 5015.519705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5015.521685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5015.525377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.526896] Lustre: Skipped 1 previous similar message [ 5018.080129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5018.080478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5023.199933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.201496] Lustre: Skipped 2 previous similar messages [ 5028.319088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5028.379834] Lustre: server umount lustre-MDT0000 complete [ 5029.665194] LustreError: 255449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535093 with bad export cookie 10815098779504626107 [ 5029.668459] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5029.670184] LustreError: 255449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5029.790696] Lustre: server umount lustre-MDT0001 complete [ 5033.605058] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5034.593177] Key type lgssc unregistered [ 5034.708380] LNet: 260118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5034.710478] LNetError: 260118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5034.719310] LNet: Removed LNI 192.168.201.103@tcp [ 5035.005339] Key type .llcrypt unregistered [ 5035.006408] Key type ._llcrypt unregistered [ 5043.758898] Key type ._llcrypt registered [ 5043.760285] Key type .llcrypt registered [ 5043.794400] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5044.137206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5044.196492] alg: No test for adler32 (adler32-zlib) [ 5045.041803] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5045.120460] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5046.695144] Key type lgssc registered [ 5047.042040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5049.486769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5049.492042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5050.615422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.833576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5054.464648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.587714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5055.800429] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5057.916611] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5059.608647] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5061.635790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5061.736631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5063.447098] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5065.653219] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5076.072909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5085.664165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.666124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.670086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.690268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.690517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.693692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.694725] Lustre: Skipped 1 previous similar message [ 5089.166404] Lustre: server umount lustre-OST0000 complete [ 5090.784041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5090.787112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.792298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5096.597943] Lustre: server umount lustre-MDT0000 complete [ 5096.928616] LustreError: 261510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.933799] LustreError: 261510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5097.859118] LustreError: 261490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535161 with bad export cookie 12621557814587759909 [ 5097.862086] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5097.862336] LustreError: 261490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5097.981093] Lustre: server umount lustre-MDT0001 complete [ 5101.588063] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5102.580315] Key type lgssc unregistered [ 5102.707539] LNet: 265832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5102.710180] LNetError: 265832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5102.717397] LNet: Removed LNI 192.168.201.103@tcp [ 5103.029032] Key type .llcrypt unregistered [ 5103.030056] Key type ._llcrypt unregistered [ 5110.187062] Key type ._llcrypt registered [ 5110.188184] Key type .llcrypt registered [ 5110.228984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5110.541936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5110.550278] alg: No test for adler32 (adler32-zlib) [ 5111.420020] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5111.517714] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5113.103142] Key type lgssc registered [ 5113.481048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5116.153913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5116.160323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.304570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.639855] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5121.568189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.725605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5123.187622] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5125.629239] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.513597] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5129.784063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.917273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5131.876172] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5134.290078] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5143.270094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5148.641566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.642983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5148.646581] Lustre: Skipped 1 previous similar message [ 5153.761821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5153.763661] Lustre: Skipped 2 previous similar messages [ 5154.640393] Lustre: server umount lustre-OST0000 complete [ 5157.855817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5157.858033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.862347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5158.881265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5164.000068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5164.002170] Lustre: Skipped 3 previous similar messages [ 5169.120288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5169.122633] Lustre: Skipped 1 previous similar message [ 5170.143147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5170.200260] Lustre: server umount lustre-MDT0000 complete [ 5171.428266] LustreError: 267011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535235 with bad export cookie 11580302416046765772 [ 5171.431743] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5171.433599] LustreError: 267011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5171.546624] Lustre: server umount lustre-MDT0001 complete [ 5175.319674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5176.378733] Key type lgssc unregistered [ 5176.520470] LNet: 270724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5176.523163] LNetError: 270724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5176.530438] LNet: Removed LNI 192.168.201.103@tcp [ 5176.884236] Key type .llcrypt unregistered [ 5176.885425] Key type ._llcrypt unregistered [ 5186.469367] Key type ._llcrypt registered [ 5186.470426] Key type .llcrypt registered [ 5186.505745] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5186.796791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.804952] alg: No test for adler32 (adler32-zlib) [ 5187.648702] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5187.727950] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5189.303148] Key type lgssc registered [ 5189.632461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5191.992320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5191.997851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.126253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.414200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5196.988055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5197.094077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5198.292588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5200.450569] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5202.082961] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5204.158846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.263200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.027517] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5208.164171] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5218.598478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5231.073103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5231.075382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.079301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.120305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5233.123715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.129607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.145090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.147333] Lustre: Skipped 1 previous similar message [ 5237.073798] Lustre: server umount lustre-OST0000 complete [ 5239.264942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.265842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.268754] Lustre: Skipped 1 previous similar message [ 5239.273226] Lustre: Skipped 1 previous similar message [ 5244.384144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.386763] Lustre: Skipped 1 previous similar message [ 5244.508455] Lustre: server umount lustre-MDT0000 complete [ 5245.729264] LustreError: 272142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535309 with bad export cookie 7702119029674157772 [ 5245.731875] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.732706] LustreError: 272142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5245.844908] Lustre: server umount lustre-MDT0001 complete [ 5249.406226] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5250.430536] Key type lgssc unregistered [ 5250.568354] LNet: 276439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5250.571983] LNetError: 276439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5250.587321] LNet: Removed LNI 192.168.201.103@tcp [ 5250.908714] Key type .llcrypt unregistered [ 5250.909871] Key type ._llcrypt unregistered [ 5257.973000] Key type ._llcrypt registered [ 5257.973896] Key type .llcrypt registered [ 5258.006343] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5258.285523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5258.314061] alg: No test for adler32 (adler32-zlib) [ 5259.162393] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5259.247945] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5260.831130] Key type lgssc registered [ 5261.164952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5263.639561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5263.645271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5264.816405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.150037] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5268.966465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.097795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5270.386996] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5272.690434] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5274.555053] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5276.879465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5276.993522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5278.888305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5281.213517] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5289.322449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 5299.684528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.685423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5299.689172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5299.690829] LustreError: Skipped 1 previous similar message [ 5299.693513] Lustre: Skipped 1 previous similar message [ 5301.395166] Lustre: server umount lustre-OST0000 complete [ 5304.800873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5304.801284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.802103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5304.810314] Lustre: Skipped 2 previous similar messages [ 5308.832723] Lustre: server umount lustre-MDT0000 complete [ 5309.920335] LustreError: 277666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5309.927371] LustreError: 277666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5310.111358] LustreError: 279519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535374 with bad export cookie 10428644727492728946 [ 5310.115450] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5310.229835] Lustre: server umount lustre-MDT0001 complete [ 5313.981927] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5315.007121] Key type lgssc unregistered [ 5315.125356] LNet: 281370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5315.128273] LNetError: 281370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5315.136236] LNet: Removed LNI 192.168.201.103@tcp [ 5315.397516] Key type .llcrypt unregistered [ 5315.398452] Key type ._llcrypt unregistered [ 5318.651966] Key type ._llcrypt registered [ 5318.652955] Key type .llcrypt registered [ 5318.687862] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 5324.331319] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5330.788512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5331.127506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5331.160082] alg: No test for adler32 (adler32-zlib) [ 5332.013950] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5332.114835] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5333.695154] Key type lgssc registered [ 5334.059585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5345.535835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5345.540744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.622457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5346.633365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5346.676650] Lustre: lustre-MDT0000: new disk, initializing [ 5346.699407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5346.707080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5348.096744] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5352.338845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.388296] Lustre: 284894:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5352.413427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5352.416173] Lustre: Skipped 1 previous similar message [ 5352.455960] Lustre: lustre-MDT0001: new disk, initializing [ 5352.490370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5352.502079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5352.505330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5353.855914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5357.519853] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5359.430674] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5361.743383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5361.855833] Lustre: lustre-OST0000: new disk, initializing [ 5361.859083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5361.863250] Lustre: 286860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5361.893489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5363.944850] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5367.603796] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5368.996895] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5369.335896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5369.340307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5369.369216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5370.085930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5371.457033] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5371.530415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5374.433985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5374.434141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.436389] LustreError: Skipped 1 previous similar message [ 5374.439995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5374.442280] Lustre: Skipped 1 previous similar message [ 5378.259906] Lustre: server umount lustre-OST0000 complete [ 5379.552401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.552921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5379.556182] Lustre: Skipped 2 previous similar messages [ 5384.672128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5384.674549] Lustre: Skipped 2 previous similar messages [ 5385.629451] Lustre: server umount lustre-MDT0000 complete [ 5386.903230] LustreError: 286859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535450 with bad export cookie 13023617883528310467 [ 5386.907116] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.907520] LustreError: 286859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5387.018646] Lustre: server umount lustre-MDT0001 complete [ 5389.857853] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:44:13 (1782535453) [ 5392.090930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.259415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.594405] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5396.430170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.549497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5397.929978] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5400.466834] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5402.378600] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5404.732698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.858399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.899723] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5409.165794] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5427.681778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.683132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.683169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.685869] Lustre: Skipped 1 previous similar message [ 5427.691962] LustreError: Skipped 1 previous similar message [ 5428.819831] Lustre: server umount lustre-OST0000 complete [ 5432.287813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5432.291300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.296962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.298445] Lustre: Skipped 1 previous similar message [ 5436.307850] Lustre: server umount lustre-MDT0000 complete [ 5437.524128] LustreError: 291773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535501 with bad export cookie 13023617883528311335 [ 5437.527607] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.527782] LustreError: 290011:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 5437.527938] LustreError: 291773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5437.635240] Lustre: server umount lustre-MDT0001 complete [ 5441.222190] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5442.269466] Key type lgssc unregistered [ 5442.404404] LNet: 292828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5442.406839] LNetError: 292828:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5442.415342] LNet: Removed LNI 192.168.201.103@tcp [ 5442.724513] Key type .llcrypt unregistered [ 5442.725981] Key type ._llcrypt unregistered [ 5451.655976] Key type ._llcrypt registered [ 5451.657522] Key type .llcrypt registered [ 5451.700871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5458.047746] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5458.450840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5458.459279] alg: No test for adler32 (adler32-zlib) [ 5459.302939] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5459.385234] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5460.959108] Key type lgssc registered [ 5461.269407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5475.480900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5475.485167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.555254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5476.565046] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5476.602307] Lustre: lustre-MDT0000: new disk, initializing [ 5476.621633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.627287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5477.830303] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5481.669245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.702875] Lustre: 296622:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5481.731389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5481.734292] Lustre: Skipped 1 previous similar message [ 5481.783972] Lustre: lustre-MDT0001: new disk, initializing [ 5481.816247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.831208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5481.835824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.114683] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5486.581200] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5488.249207] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5490.200054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5490.286271] Lustre: lustre-OST2710: new disk, initializing [ 5490.287673] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5490.290218] Lustre: 298591:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5490.314820] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5492.206032] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5492.980382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5492.982770] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5493.008181] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5495.542628] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5497.614179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5497.669418] Lustre: lustre-OST03e8: new disk, initializing [ 5497.671246] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5497.674388] Lustre: 299653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5497.705152] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5499.349107] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5501.539306] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5501.692496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5501.694947] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5501.728993] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5504.089332] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5504.161714] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5505.629726] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5505.709260] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5507.132535] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5507.208846] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5508.543982] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5508.612644] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5512.162430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.164568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5512.169178] Lustre: Skipped 1 previous similar message [ 5512.173831] Lustre: Skipped 3 previous similar messages [ 5515.751440] Lustre: server umount lustre-MDT0000 complete [ 5516.972169] LustreError: 296615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535580 with bad export cookie 10474664011169696457 [ 5516.974738] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.975777] LustreError: 296615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5517.091516] Lustre: server umount lustre-MDT0001 complete [ 5528.577625] Lustre: server umount lustre-OST2710 complete [ 5540.063053] Lustre: server umount lustre-OST03e8 complete [ 5546.020872] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5549.419415] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5561.466358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.548813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5561.558595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5561.594873] Lustre: lustre-MDT0000: new disk, initializing [ 5561.615117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.620406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5562.946568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5567.171246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.207990] Lustre: 304893:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5567.317814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5568.752471] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5572.501810] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5574.393460] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5576.741615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5576.846418] Lustre: lustre-OST0000: new disk, initializing [ 5576.848704] Lustre: Skipped 1 previous similar message [ 5576.851378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5576.853582] Lustre: Skipped 2 previous similar messages [ 5576.855324] Lustre: 306862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5576.883560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.886493] Lustre: Skipped 1 previous similar message [ 5578.862704] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5580.862227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5580.866354] Lustre: Skipped 1 previous similar message [ 5580.869074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5580.909375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5582.590827] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5583.991626] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5584.062174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5585.494054] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5585.572641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5586.400226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.403643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.409891] Lustre: Skipped 2 previous similar messages [ 5586.412277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5588.304659] Lustre: server umount lustre-OST0000 complete [ 5591.008871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.009585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5591.012129] Lustre: Skipped 1 previous similar message [ 5591.015946] Lustre: Skipped 1 previous similar message [ 5595.678623] Lustre: server umount lustre-MDT0000 complete [ 5596.886458] LustreError: 305691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535660 with bad export cookie 10474664011169698123 [ 5596.890300] LustreError: 305691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5596.890527] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.420553] Lustre: server umount lustre-MDT0001 complete [ 5609.297961] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:47:52 (1782535672) [ 5609.821906] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5610.401490] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:47:54 (1782535674) [ 5611.425498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5620.519331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.599811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5620.608723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5620.663216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5622.015854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5624.884790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5624.913338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5624.925105] Lustre: 310711:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5626.500857] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5642.402329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5642.482711] Lustre: Denying initial registration attempt from nid 192.168.201.103@tcp, specified as failover [ 5642.486149] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5642.489368] LustreError: 312162:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 5642.491990] LustreError: 312162:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5642.494417] LustreError: 312162:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5642.520646] Lustre: server umount lustre-OST0000 complete [ 5642.522780] LustreError: 312162:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5645.792378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5645.795434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5645.800384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.936210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.939401] Lustre: Skipped 3 previous similar messages [ 5659.103111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5659.159185] Lustre: server umount lustre-MDT0000 complete [ 5660.434294] LustreError: 310702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535724 with bad export cookie 10474664011169698991 [ 5660.437757] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.439499] LustreError: 310702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5660.560415] Lustre: server umount lustre-MDT0001 complete [ 5664.086091] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5665.146701] Key type lgssc unregistered [ 5665.278425] LNet: 313282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5665.281748] LNetError: 313282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5665.289283] LNet: Removed LNI 192.168.201.103@tcp [ 5665.586835] Key type .llcrypt unregistered [ 5665.587920] Key type ._llcrypt unregistered [ 5669.180145] Key type ._llcrypt registered [ 5669.181165] Key type .llcrypt registered [ 5669.218747] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5669.570879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5669.607053] alg: No test for adler32 (adler32-zlib) [ 5670.461402] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5670.557099] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5672.135115] Key type lgssc registered [ 5672.502597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5696.584542] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5699.486553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5699.494542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.597813] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5700.604956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5700.667533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.131056] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5704.973973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.009836] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5705.021352] Lustre: 316021:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5705.120909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5706.470948] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5708.919300] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5710.732487] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5726.049973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5726.120409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5726.191046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5728.157359] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5730.571334] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5731.334731] Lustre: server umount lustre-OST0000 complete [ 5735.411529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.412464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5735.416169] Lustre: Skipped 1 previous similar message [ 5738.770038] Lustre: server umount lustre-MDT0000 complete [ 5740.037433] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535804 with bad export cookie 17300271322034400604 [ 5740.040583] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.042093] LustreError: 316762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5740.168991] Lustre: server umount lustre-MDT0001 complete [ 5744.019974] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5745.061513] Key type lgssc unregistered [ 5745.198379] LNet: 319583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5745.201385] LNetError: 319583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5745.210320] LNet: Removed LNI 192.168.201.103@tcp [ 5745.522470] Key type .llcrypt unregistered [ 5745.523989] Key type ._llcrypt unregistered [ 5754.611445] Key type ._llcrypt registered [ 5754.612362] Key type .llcrypt registered [ 5754.646068] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5754.963238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5755.012986] alg: No test for adler32 (adler32-zlib) [ 5755.872118] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5755.964389] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5757.543114] Key type lgssc registered [ 5757.912180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5760.497356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5760.502888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5761.640119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.002684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5765.784883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.926682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5767.335669] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5769.694643] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5771.571127] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5775.970518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.092228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.080160] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5780.561023] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5795.810441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5795.811902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5795.815958] Lustre: Skipped 2 previous similar messages [ 5795.820308] Lustre: Skipped 2 previous similar messages [ 5797.123682] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 5800.152682] Lustre: server umount lustre-MDT0000 complete [ 5800.928712] LustreError: 321001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5800.933189] LustreError: 321001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5801.408363] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535865 with bad export cookie 7061192503173626438 [ 5801.411722] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5801.412506] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5802.235756] LustreError: 320972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.244508] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 5806.047833] LustreError: 320971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.048159] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.060190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5807.604146] Lustre: server umount lustre-MDT0001 complete [ 5809.732975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5812.537292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.527583] LustreError: 320287:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cf82ff3d x1869124278968320/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5826.639527] LustreError: 322806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.647524] LustreError: 322806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5826.666052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5826.686635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5828.121393] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5831.026930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.155848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5832.541599] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5834.936805] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5836.262032] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5838.571404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5840.363650] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5840.924253] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5846.496580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5846.498527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5846.500452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5846.506138] Lustre: Skipped 1 previous similar message [ 5846.509089] Lustre: Skipped 2 previous similar messages [ 5848.276973] Lustre: server umount lustre-OST0000 complete [ 5851.617157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5851.617404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.623031] Lustre: Skipped 1 previous similar message [ 5855.701673] Lustre: server umount lustre-MDT0000 complete [ 5856.736181] LustreError: 324591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.741565] LustreError: 324591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5856.954409] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535920 with bad export cookie 7061192503173633592 [ 5856.957985] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5856.959884] LustreError: 320958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5857.068202] Lustre: server umount lustre-MDT0001 complete [ 5860.701656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5861.742403] Key type lgssc unregistered [ 5861.883361] LNet: 327259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5861.885482] LNetError: 327259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5861.892286] LNet: Removed LNI 192.168.201.103@tcp [ 5862.204261] Key type .llcrypt unregistered [ 5862.205379] Key type ._llcrypt unregistered [ 5871.718609] Key type ._llcrypt registered [ 5871.720155] Key type .llcrypt registered [ 5871.761422] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5872.116296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5872.178578] alg: No test for adler32 (adler32-zlib) [ 5873.032740] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 5873.117258] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5874.695164] Key type lgssc registered [ 5875.068215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5877.690122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5877.698582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.850421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.269332] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5883.200044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.340826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.714695] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5887.203960] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5889.032631] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5893.152990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.158809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.161488] Lustre: Skipped 1 previous similar message [ 5898.208056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5898.209724] Lustre: Skipped 1 previous similar message [ 5903.328268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.331058] Lustre: Skipped 1 previous similar message [ 5904.351131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5904.408537] Lustre: server umount lustre-MDT0000 complete [ 5905.671252] LustreError: 328676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782535969 with bad export cookie 5984806100772033226 [ 5905.674655] LustreError: 328676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5905.674857] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.796617] Lustre: server umount lustre-MDT0001 complete [ 5910.667601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.773465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5910.783869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5910.855959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.230748] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5915.074790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.110230] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5915.120535] Lustre: 331239:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5915.216766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.674759] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5919.062835] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5920.891796] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5925.538044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.606562] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5925.609616] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5925.614226] LustreError: 333122:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5925.617695] LustreError: 333122:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5925.620781] LustreError: 333122:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5925.646538] Lustre: server umount lustre-OST0000 complete [ 5925.648350] LustreError: 333122:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5930.402516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5930.471850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5930.533545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5932.392230] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5934.716475] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5937.201732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.241405] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5937.271275] Lustre: lustre-OST0001: new disk, initializing [ 5937.273625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5937.277146] Lustre: 334652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5938.889894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5938.893801] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5938.921594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5938.943613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5939.294390] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5941.725211] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5944.291891] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.292157] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.294080] LustreError: Skipped 1 previous similar message [ 5944.297249] Lustre: Skipped 1 previous similar message [ 5944.299105] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5944.301095] Lustre: Skipped 1 previous similar message [ 5948.795325] Lustre: server umount lustre-OST0001 complete [ 5949.416253] LustreError: 333589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.420485] LustreError: 333589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5954.530806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.530927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.531387] LustreError: 333587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.532873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.532989] LustreError: Skipped 1 previous similar message [ 5954.536207] Lustre: Skipped 1 previous similar message [ 5956.167979] Lustre: server umount lustre-OST0000 complete [ 5959.648775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.654130] Lustre: Skipped 1 previous similar message [ 5963.547642] Lustre: server umount lustre-MDT0000 complete [ 5964.768373] LustreError: 331249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.774039] LustreError: 331249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5964.828267] LustreError: 331231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536028 with bad export cookie 5984806100772033765 [ 5964.832605] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5964.833179] LustreError: 331231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5964.952303] Lustre: server umount lustre-MDT0001 complete [ 5968.771035] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5969.785876] Key type lgssc unregistered [ 5969.908433] LNet: 336611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5969.911875] LNetError: 336611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5969.920352] LNet: Removed LNI 192.168.201.103@tcp [ 5970.209753] Key type .llcrypt unregistered [ 5970.210929] Key type ._llcrypt unregistered [ 6000.420476] Key type ._llcrypt registered [ 6000.421586] Key type .llcrypt registered [ 6000.458964] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6006.620368] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6007.032876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6007.054334] alg: No test for adler32 (adler32-zlib) [ 6007.905837] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6007.993131] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6009.567111] Key type lgssc registered [ 6009.912367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6021.064294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6021.071146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.148315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6022.159537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6022.199897] Lustre: lustre-MDT0000: new disk, initializing [ 6022.222160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.228415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6023.599276] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6027.683317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.730382] Lustre: 341511:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6027.752562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6027.754223] Lustre: Skipped 1 previous similar message [ 6027.791708] Lustre: lustre-MDT0001: new disk, initializing [ 6027.827120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6027.842382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6027.846840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6029.179788] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6032.851434] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6034.608640] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6036.805399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6036.903018] Lustre: lustre-OST0000: new disk, initializing [ 6036.904692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6036.907472] Lustre: 343480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6036.934752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.072192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6038.076031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6038.104031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6038.808514] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6042.480786] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6043.919425] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6043.991041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6045.368476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6045.443410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6048.226779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.227398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6048.231748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6048.233126] LustreError: Skipped 1 previous similar message [ 6052.174605] Lustre: server umount lustre-OST0000 complete [ 6058.465070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.465186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.465951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.465955] Lustre: Skipped 1 previous similar message [ 6058.474516] Lustre: Skipped 2 previous similar messages [ 6059.544197] Lustre: server umount lustre-MDT0000 complete [ 6060.758179] LustreError: 342307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536124 with bad export cookie 1423431881091896476 [ 6060.762935] LustreError: 342307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6060.762955] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6060.895979] Lustre: server umount lustre-MDT0001 complete [ 6063.632733] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:55:27 (1782536127) [ 6066.091391] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:55:29 (1782536129) [ 6068.869860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.031576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.400108] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6073.310322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.438274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6074.783665] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6077.092183] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6078.964629] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6081.213089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6083.310200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6085.690903] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6099.525598] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6100.094836] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6100.625811] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6101.154368] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6101.780570] Lustre: Failing over lustre-MDT0000 [ 6101.971495] Lustre: server umount lustre-MDT0000 complete [ 6104.031994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6104.034765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.039301] LustreError: 346232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6104.046090] LustreError: 346232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6105.056955] LustreError: 346959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.062452] LustreError: 346959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6106.369446] LustreError: 348726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.176057] LustreError: 346233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.181379] LustreError: 346233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6114.812834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6114.855289] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6114.947450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6114.949530] Lustre: Skipped 1 previous similar message [ 6114.982463] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6116.366656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6116.603395] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6120.423200] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6120.432792] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6120.449754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6122.322607] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6122.876327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6124.725408] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 6125.377936] Lustre: Failing over lustre-MDT0000 [ 6125.535880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6125.537224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.541220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.541989] Lustre: Skipped 3 previous similar messages [ 6125.544735] Lustre: Skipped 1 previous similar message [ 6125.649600] Lustre: server umount lustre-MDT0000 complete [ 6126.848910] LustreError: 348726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.853646] LustreError: 348726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6135.776585] LustreError: 346959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.783491] LustreError: 346959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6138.549821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.589448] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.673816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.715414] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6139.994601] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6142.203437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6143.973788] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6143.976505] Lustre: Skipped 2 previous similar messages [ 6143.988931] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6144.007115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6145.706840] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6146.246799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6149.089854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6149.090333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.091198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.091207] Lustre: Skipped 2 previous similar messages [ 6149.098740] Lustre: Skipped 2 previous similar messages [ 6154.208665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6154.210883] Lustre: Skipped 4 previous similar messages [ 6154.579214] Lustre: server umount lustre-MDT0000 complete [ 6155.804134] LustreError: 346214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536219 with bad export cookie 1423431881091899157 [ 6155.807761] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6155.808050] LustreError: 346214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6155.924048] Lustre: server umount lustre-MDT0001 complete [ 6167.448262] Lustre: server umount lustre-OST0000 complete [ 6172.463147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.609812] LustreError: 352579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.614518] LustreError: 352579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6172.636100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.030283] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6176.937507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.441965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6180.892716] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6182.742430] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6184.972519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6187.107369] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6189.600702] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6197.417633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6202.850964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.852492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6202.855317] Lustre: Skipped 1 previous similar message [ 6202.858229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.861134] Lustre: Skipped 1 previous similar message [ 6203.281472] Lustre: server umount lustre-OST0000 complete [ 6206.622335] Lustre: server umount lustre-MDT0000 complete [ 6207.884287] LustreError: 352562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536271 with bad export cookie 1423431881091899878 [ 6207.887737] LustreError: 352562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6207.888931] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6207.990396] Lustre: server umount lustre-MDT0001 complete [ 6211.787682] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6212.916960] Key type lgssc unregistered [ 6213.047533] LNet: 356128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6213.051303] LNetError: 356128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6213.059470] LNet: Removed LNI 192.168.201.103@tcp [ 6213.427909] Key type .llcrypt unregistered [ 6213.428993] Key type ._llcrypt unregistered [ 6222.768239] Key type ._llcrypt registered [ 6222.769380] Key type .llcrypt registered [ 6222.808299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6223.246808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.259074] alg: No test for adler32 (adler32-zlib) [ 6224.106326] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6224.189091] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6225.767121] Key type lgssc registered [ 6226.121717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6228.609248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6228.615591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.754224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.071338] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6233.838388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6233.961705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6235.302024] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6237.559959] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6239.341299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6241.632071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.746583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.622923] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6246.024054] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6254.060976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6264.289617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.291072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6264.294293] Lustre: Skipped 2 previous similar messages [ 6264.297014] Lustre: Skipped 2 previous similar messages [ 6266.289531] Lustre: server umount lustre-MDT0000 complete [ 6267.584540] LustreError: 357644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536331 with bad export cookie 7963508445346919203 [ 6267.589386] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.715731] Lustre: server umount lustre-MDT0001 complete [ 6279.257783] Lustre: server umount lustre-OST0000 complete [ 6283.653919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.827071] LustreError: 361350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.835952] LustreError: 361350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6283.855379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.252862] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6288.449297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.584176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.963991] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6292.244244] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6293.927599] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6296.096630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.071335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6300.477804] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6308.522543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6319.073555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.075263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6319.077579] Lustre: Skipped 2 previous similar messages [ 6319.079177] Lustre: Skipped 1 previous similar message [ 6324.061893] Lustre: server umount lustre-MDT0000 complete [ 6324.192613] LustreError: 361352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.197893] LustreError: 361352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6325.257121] LustreError: 361332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536389 with bad export cookie 7963508445346920302 [ 6325.260235] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6325.263175] LustreError: 361332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 6325.377660] Lustre: server umount lustre-MDT0001 complete [ 6336.868811] Lustre: server umount lustre-OST0000 complete [ 6341.981029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.149170] LustreError: 365278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.172563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.174647] Lustre: Skipped 1 previous similar message [ 6343.572595] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6346.385677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.983769] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6350.453453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6352.232691] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6354.493280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.593301] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6358.946233] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6367.970560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6372.320283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6372.323634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.329443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.332435] Lustre: Skipped 1 previous similar message [ 6373.348077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.722139] Lustre: server umount lustre-OST0000 complete [ 6378.464977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6378.465705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.468482] Lustre: Skipped 2 previous similar messages [ 6378.472604] Lustre: Skipped 3 previous similar messages [ 6383.584647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.587593] Lustre: Skipped 1 previous similar message [ 6392.287119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6392.349811] Lustre: server umount lustre-MDT0000 complete [ 6393.576780] LustreError: 367777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536457 with bad export cookie 7963508445346921310 [ 6393.580521] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6393.580795] LustreError: 367777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6393.701634] Lustre: server umount lustre-MDT0001 complete [ 6397.413199] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6398.398162] Key type lgssc unregistered [ 6398.514318] LNet: 368840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6398.516916] LNetError: 368840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6398.523250] LNet: Removed LNI 192.168.201.103@tcp [ 6398.832656] Key type .llcrypt unregistered [ 6398.835298] Key type ._llcrypt unregistered [ 6408.361934] Key type ._llcrypt registered [ 6408.363782] Key type .llcrypt registered [ 6408.405602] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6408.729607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6408.762588] alg: No test for adler32 (adler32-zlib) [ 6409.607448] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6409.694043] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6411.271137] Key type lgssc registered [ 6411.642839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6414.133316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6414.138645] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6414.141566] LustreError: 370257:0:(osd_handler.c:8552:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6414.148953] LustreError: 370257:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6414.151036] LustreError: 370257:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6414.153128] LustreError: 370257:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6414.155367] LustreError: 370257:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6416.735342] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.738511] LustreError: 370654:0:(osd_handler.c:8552:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6416.745931] LustreError: 370654:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6416.748951] LustreError: 370654:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6416.753147] LustreError: 370654:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6416.757839] LustreError: 370654:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6422.914437] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6423.893834] Key type lgssc unregistered [ 6424.017340] LNet: 371654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6424.020148] LNetError: 371654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6424.029304] LNet: Removed LNI 192.168.201.103@tcp [ 6424.317595] Key type .llcrypt unregistered [ 6424.318573] Key type ._llcrypt unregistered [ 6430.270069] Key type ._llcrypt registered [ 6430.271822] Key type .llcrypt registered [ 6430.311246] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6436.638147] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6436.938214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6436.950516] alg: No test for adler32 (adler32-zlib) [ 6437.806659] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6437.906524] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6439.487151] Key type lgssc registered [ 6439.837354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6451.022943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6451.029060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.117747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6452.127726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6452.164454] Lustre: lustre-MDT0000: new disk, initializing [ 6452.184066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.190652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6453.448682] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6457.509433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6457.549597] Lustre: 374782:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6457.588919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6457.591626] Lustre: Skipped 1 previous similar message [ 6457.632260] Lustre: lustre-MDT0001: new disk, initializing [ 6457.663134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6457.674786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6457.677725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.024586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6462.696944] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6464.508587] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6466.744605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6466.852084] Lustre: lustre-OST0000: new disk, initializing [ 6466.853844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6466.856419] Lustre: 376750:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6466.885794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.227108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6467.231251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6467.271499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6468.855608] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6472.529474] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.902817] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.977930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6475.319342] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6475.382856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6477.795268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.796397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.796512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6477.801589] Lustre: Skipped 1 previous similar message [ 6477.808640] LustreError: Skipped 1 previous similar message [ 6482.128810] Lustre: server umount lustre-OST0000 complete [ 6483.423889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6483.425932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.429583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6483.431014] Lustre: Skipped 1 previous similar message [ 6488.032213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6488.032918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.496046] Lustre: server umount lustre-MDT0000 complete [ 6490.698180] LustreError: 375579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536554 with bad export cookie 3116775270558533025 [ 6490.701653] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6490.703048] LustreError: 375579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6490.816713] Lustre: server umount lustre-MDT0001 complete [ 6493.491986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6494.017687] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:02:37 (1782536557) [ 6496.156677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.308448] LustreError: 379209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.313440] LustreError: 379209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6496.329155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.624083] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6500.379171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.500232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6501.813884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6504.084841] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6505.828353] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6508.004541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.068170] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6512.354930] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6514.570854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6514.633311] Lustre: lustre-OST0001: new disk, initializing [ 6514.635373] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6514.638927] Lustre: 382045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6514.669889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6514.673391] Lustre: Skipped 1 previous similar message [ 6516.666103] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6520.384889] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6521.344105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6521.347924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6521.377523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6526.433298] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6526.433632] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.435288] LustreError: Skipped 1 previous similar message [ 6526.442516] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6526.445638] Lustre: Skipped 2 previous similar messages [ 6528.447129] Lustre: server umount lustre-OST0001 complete [ 6529.082887] LustreError: 382764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.581549] LustreError: 381031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.589794] LustreError: 381031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6537.699272] LustreError: 382764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.704802] LustreError: 382764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6541.279820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.282388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.285739] Lustre: Skipped 1 previous similar message [ 6541.287438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6541.288911] Lustre: Skipped 2 previous similar messages [ 6542.819220] LustreError: 381030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.826754] LustreError: 381030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6546.577467] Lustre: server umount lustre-OST0000 complete [ 6553.056205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6553.058251] Lustre: Skipped 5 previous similar messages [ 6562.271118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6562.328299] Lustre: server umount lustre-MDT0000 complete [ 6563.296062] LustreError: 379210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.302366] LustreError: 379210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6563.503154] LustreError: 379190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536627 with bad export cookie 3116775270558533879 [ 6563.505921] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.506493] LustreError: 379190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6569.711137] Lustre: server umount lustre-MDT0001 complete [ 6573.209907] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6574.168217] Key type lgssc unregistered [ 6574.298280] LNet: 384027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6574.301462] LNetError: 384027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6574.311237] LNet: Removed LNI 192.168.201.103@tcp [ 6574.585576] Key type .llcrypt unregistered [ 6574.586553] Key type ._llcrypt unregistered [ 6591.843651] Key type ._llcrypt registered [ 6591.844706] Key type .llcrypt registered [ 6591.887388] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6592.197374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6592.214476] alg: No test for adler32 (adler32-zlib) [ 6593.069557] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6593.163181] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6594.735135] Key type lgssc registered [ 6595.088903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6597.469950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6597.475860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.553351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6598.559103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6598.618526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.872330] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6602.561368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.594343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6602.607251] Lustre: 386799:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.699675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6603.960749] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6606.137692] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6607.882963] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6610.400695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.401448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.404328] Lustre: Skipped 1 previous similar message [ 6614.676143] Lustre: server umount lustre-MDT0000 complete [ 6615.897549] LustreError: 386792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536679 with bad export cookie 9014180195371725489 [ 6615.900480] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.902904] LustreError: 386792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6616.010155] Lustre: server umount lustre-MDT0001 complete [ 6619.141415] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:04:42 (1782536682) [ 6621.212988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.364718] LustreError: 389306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6621.369014] LustreError: 389306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6621.382875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6622.709959] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6625.346813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.463104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.799703] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6629.091580] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6630.752164] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6633.174067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.271110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6633.331759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.163884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6637.525520] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6641.163226] Lustre: Permanently deactivating lustre-OST0000 [ 6641.168756] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6641.287354] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6641.758947] LustreError: 392095:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6641.762209] LustreError: 392095:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6641.764445] LustreError: 392095:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6642.332658] LustreError: 392144:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6642.335060] LustreError: 392144:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6649.038059] Lustre: server umount lustre-OST0000 complete [ 6649.912324] LustreError: 392393:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6649.915580] LustreError: 392393:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6650.848144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.848762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6650.852351] Lustre: Skipped 1 previous similar message [ 6650.854442] Lustre: Skipped 1 previous similar message [ 6655.968616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6655.971516] Lustre: Skipped 2 previous similar messages [ 6660.087507] Lustre: server umount lustre-MDT0000 complete [ 6661.088022] LustreError: 389307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6661.091845] LustreError: 389307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6661.292078] LustreError: 389286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536725 with bad export cookie 9014180195371726056 [ 6661.295098] LustreError: 389286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6661.295175] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.890253] Lustre: server umount lustre-MDT0001 complete [ 6674.609132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.895699] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6676.851338] LustreError: 393550:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6679.038863] Lustre: 393889:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.103@tcp [ 6679.804470] Lustre: server umount MGS complete [ 6683.457492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6683.605938] LustreError: 394613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.610809] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6683.615764] Lustre: Skipped 1 previous similar message [ 6683.634425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6684.936751] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6687.731183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.827156] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6689.436773] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6691.689811] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6693.636518] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6694.264347] Lustre: Permanently reactivating lustre-OST0000 [ 6694.269247] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6694.271163] Lustre: Skipped 2 previous similar messages [ 6719.967101] LustreError: 396394:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6722.422777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6722.536806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6722.539749] Lustre: Skipped 1 previous similar message [ 6724.380528] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6726.643320] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6739.939351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.940138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6739.941474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6739.946081] Lustre: Skipped 1 previous similar message [ 6739.949261] LustreError: Skipped 1 previous similar message [ 6744.463244] Lustre: server umount lustre-OST0000 complete [ 6749.151818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6749.153753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6749.157099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6749.158597] Lustre: Skipped 1 previous similar message [ 6751.769448] Lustre: server umount lustre-MDT0000 complete [ 6752.965142] LustreError: 394596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536816 with bad export cookie 9014180195371727078 [ 6752.968097] LustreError: 394596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6752.968226] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6753.072410] Lustre: server umount lustre-MDT0001 complete [ 6756.415650] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6757.420843] Key type lgssc unregistered [ 6757.534305] LNet: 399284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6757.536421] LNetError: 399284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6757.543289] LNet: Removed LNI 192.168.201.103@tcp [ 6757.868669] Key type .llcrypt unregistered [ 6757.869601] Key type ._llcrypt unregistered [ 6763.468872] Key type ._llcrypt registered [ 6763.469825] Key type .llcrypt registered [ 6763.503060] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6769.420290] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6769.759448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6769.792655] alg: No test for adler32 (adler32-zlib) [ 6770.633037] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6770.718371] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6772.295121] Key type lgssc registered [ 6772.665121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6783.629828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6783.635256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.726682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6784.736665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6784.782908] Lustre: lustre-MDT0000: new disk, initializing [ 6784.807196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.815914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6786.244068] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6790.210046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.249931] Lustre: 402410:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6790.284708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6790.287109] Lustre: Skipped 1 previous similar message [ 6790.321963] Lustre: lustre-MDT0001: new disk, initializing [ 6790.353356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6790.364993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6790.369541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6791.640095] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6795.182093] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6796.892802] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6799.161114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.267075] Lustre: lustre-OST0000: new disk, initializing [ 6799.268572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6799.270973] Lustre: 404381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6799.300453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.195427] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6804.723080] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6806.003354] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6806.516964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6806.519892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6806.550772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6807.086670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6808.315154] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6808.376609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6811.616764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6811.618381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.619089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.620065] LustreError: Skipped 1 previous similar message [ 6811.623019] Lustre: Skipped 1 previous similar message [ 6815.052996] Lustre: server umount lustre-OST0000 complete [ 6816.736096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.740514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6816.742086] Lustre: Skipped 1 previous similar message [ 6821.344139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6821.348484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.355845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6822.358645] Lustre: server umount lustre-MDT0000 complete [ 6823.588028] LustreError: 402404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536887 with bad export cookie 11007357287268048356 [ 6823.591958] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.592321] LustreError: 402404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6823.706142] Lustre: server umount lustre-MDT0001 complete [ 6826.288166] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6826.796521] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:08:10 (1782536890) [ 6828.945099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.091857] LustreError: 406843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.097116] LustreError: 406843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6829.115830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.384485] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6833.075955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6833.205600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6834.528995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6836.840077] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6838.580524] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6840.786524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.796345] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6845.122289] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6845.743662] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6856.516945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6859.746646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6859.747268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.750071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6859.753256] Lustre: Skipped 1 previous similar message [ 6859.755685] Lustre: Skipped 1 previous similar message [ 6864.077480] Lustre: server umount lustre-OST0000 complete [ 6868.959889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6868.962106] LustreError: Skipped 1 previous similar message [ 6868.963315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.966954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6868.968521] Lustre: Skipped 1 previous similar message [ 6871.506091] Lustre: server umount lustre-MDT0000 complete [ 6872.697112] LustreError: 406825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536936 with bad export cookie 11007357287268049238 [ 6872.699687] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.701562] LustreError: 406825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6872.814950] Lustre: server umount lustre-MDT0001 complete [ 6876.508565] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6877.684457] Key type lgssc unregistered [ 6877.834506] LNet: 410584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6877.837974] LNetError: 410584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6877.846489] LNet: Removed LNI 192.168.201.103@tcp [ 6878.225218] Key type .llcrypt unregistered [ 6878.226211] Key type ._llcrypt unregistered [ 6886.086916] Key type ._llcrypt registered [ 6886.087939] Key type .llcrypt registered [ 6886.123448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6886.235434] Key type .llcrypt unregistered [ 6886.237120] Key type ._llcrypt unregistered [ 6893.351100] Key type ._llcrypt registered [ 6893.352754] Key type .llcrypt registered [ 6893.397166] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6893.876823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6893.883758] alg: No test for adler32 (adler32-zlib) [ 6894.757494] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6894.851529] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6896.431117] Key type lgssc registered [ 6896.756297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6899.332543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6899.345290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.503131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6901.848494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6905.080428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6905.190114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6907.008163] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6909.656970] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6910.307831] LustreError: 412768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6910.314760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6910.328398] LustreError: 412768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6912.155632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.273411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.547098] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6922.727541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.732072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.739910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.743313] Lustre: Skipped 1 previous similar message [ 6927.019397] Lustre: server umount lustre-OST0000 complete [ 6930.916167] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6930.920870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6930.928660] Lustre: Skipped 1 previous similar message [ 6930.932032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.961336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6932.962472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.746206] Lustre: server umount lustre-MDT0000 complete [ 6935.954295] LustreError: 412751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782536999 with bad export cookie 9457022970523889832 [ 6935.957053] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6935.958707] LustreError: 412751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6936.081924] Lustre: server umount lustre-MDT0001 complete [ 6939.648921] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6940.596304] Key type lgssc unregistered [ 6940.718332] LNet: 415915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6940.720423] LNetError: 415915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6940.727310] LNet: Removed LNI 192.168.201.103@tcp [ 6941.043110] Key type .llcrypt unregistered [ 6941.046549] Key type ._llcrypt unregistered [ 6950.312955] Key type ._llcrypt registered [ 6950.314022] Key type .llcrypt registered [ 6950.352686] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6950.721340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6950.770990] alg: No test for adler32 (adler32-zlib) [ 6951.632522] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 6951.730428] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6953.319141] Key type lgssc registered [ 6953.773379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6956.272571] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6956.279366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6971.743351] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6976.863265] LustreError: 417281:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 7003.487163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.250608] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7007.437149] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7009.611514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.779579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.804187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7011.162574] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7013.942196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.057518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7015.353602] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7017.645754] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7019.397300] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7023.586642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7023.587525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7023.590015] Lustre: Skipped 1 previous similar message [ 7027.724320] Lustre: server umount lustre-OST0000 complete [ 7029.727899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7029.729936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.733392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7029.735121] Lustre: Skipped 1 previous similar message [ 7033.825130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7033.826158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.234637] Lustre: server umount lustre-MDT0000 complete [ 7036.563391] LustreError: 417289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782537100 with bad export cookie 10608529266544685009 [ 7036.566088] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.566467] LustreError: 417289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7036.675820] Lustre: server umount lustre-MDT0001 complete [ 7040.214318] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7041.110176] Key type lgssc unregistered [ 7041.235350] LNet: 420849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7041.237391] LNetError: 420849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7041.244244] LNet: Removed LNI 192.168.201.103@tcp [ 7041.543663] Key type .llcrypt unregistered [ 7041.544625] Key type ._llcrypt unregistered [ 7050.032357] Key type ._llcrypt registered [ 7050.033210] Key type .llcrypt registered [ 7050.064915] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7050.371825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7050.426026] alg: No test for adler32 (adler32-zlib) [ 7051.271526] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 7051.352750] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7052.935154] Key type lgssc registered [ 7053.323876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7056.073088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7056.079917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.232667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.598732] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7061.681262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.812038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7063.234581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7065.471116] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7067.234421] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7069.797880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.991581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7072.018432] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7074.668697] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7083.302879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7087.583935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.587167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.590775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.609141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.610637] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7090.439681] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 7090.443221] Lustre: Skipped 2 previous similar messages [ 7092.281635] Lustre: server umount lustre-MDT0000 complete [ 7093.002106] LustreError: 422223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1782537157 with bad export cookie 13966687105128858141 [ 7093.008979] LustreError: 422223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7093.615213] LustreError: 422975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782537157 with bad export cookie 13966687105128857518 [ 7093.620531] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7093.728225] LustreError: 422241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7093.729511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.732508] LustreError: 422241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7093.737211] Lustre: Skipped 1 previous similar message [ 7093.738149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.848040] LustreError: 422968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.848597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.852100] LustreError: 422968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7099.730788] Lustre: server umount lustre-OST0000 complete [ 7102.667880] Lustre: server umount lustre-MDT0001 complete [ 7106.502436] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7107.394634] Key type lgssc unregistered [ 7107.514426] LNet: 425943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7107.517484] LNetError: 425943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7107.527327] LNet: Removed LNI 192.168.201.103@tcp [ 7107.825808] Key type .llcrypt unregistered [ 7107.826880] Key type ._llcrypt unregistered [ 7117.188084] Key type ._llcrypt registered [ 7117.188984] Key type .llcrypt registered [ 7117.223763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7117.628755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.636739] alg: No test for adler32 (adler32-zlib) [ 7118.473413] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 7118.553751] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7120.127152] Key type lgssc registered [ 7120.449112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7122.817331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7122.822423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.969322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.217356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7127.885436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7128.006706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7129.229441] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7131.298509] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7132.877183] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7134.914927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.019398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.721763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7138.876018] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7148.328898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7153.634851] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7153.637043] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7153.640668] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7153.642436] Lustre: Skipped 1 previous similar message [ 7154.434662] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 7158.753123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7158.754751] Lustre: Skipped 1 previous similar message [ 7163.874051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7163.877317] Lustre: Skipped 2 previous similar messages [ 7164.383065] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7164.430740] Lustre: server umount lustre-MDT0001 complete [ 7164.667588] LustreError: 429177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.672443] LustreError: 429177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7168.992423] LustreError: 427339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7168.993215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7168.997907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.000753] Lustre: Skipped 1 previous similar message [ 7169.002276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.046245] Lustre: server umount lustre-OST0000 complete [ 7173.319420] Lustre: server umount lustre-MDT0000 complete [ 7177.717412] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7178.595278] Key type lgssc unregistered [ 7178.712340] LNet: 431042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7178.714617] LNetError: 431042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7178.721270] LNet: Removed LNI 192.168.201.103@tcp [ 7179.022337] Key type .llcrypt unregistered [ 7179.023412] Key type ._llcrypt unregistered [ 7187.020378] Key type ._llcrypt registered [ 7187.021439] Key type .llcrypt registered [ 7187.057008] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7194.211585] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7194.572345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7194.593856] alg: No test for adler32 (adler32-zlib) [ 7195.444268] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 7195.528285] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7197.103118] Key type lgssc registered [ 7197.530742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7210.679067] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7217.251893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7217.261561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.365245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7218.376784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7218.411159] Lustre: lustre-MDT0000: new disk, initializing [ 7218.433945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7218.441303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7220.166965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7226.827037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7226.916044] Lustre: lustre-OST0000: new disk, initializing [ 7226.917658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7226.919111] Lustre: Skipped 1 previous similar message [ 7226.920728] Lustre: 436045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7226.941099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7228.953203] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7232.016236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7232.020704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7232.034926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7233.660872] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7240.410562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.470179] Lustre: 435029:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7240.507393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7240.548845] Lustre: lustre-MDT0001: new disk, initializing [ 7240.601351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7240.622101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 7240.626869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 7242.359200] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7246.537560] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7246.628487] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7247.991717] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7248.053604] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7250.709763] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 7250.916884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.917087] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 7255.805644] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 7255.813138] Lustre: Skipped 2 previous similar messages [ 7260.924305] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 7260.926945] Lustre: Skipped 2 previous similar messages [ 7264.223074] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7264.275670] Lustre: server umount lustre-MDT0001 complete [ 7271.394851] LustreError: 435063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7271.405093] LustreError: 435063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7276.511968] LustreError: 437385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.512302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7276.518793] Lustre: Skipped 1 previous similar message [ 7276.520443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7276.521968] Lustre: Skipped 2 previous similar messages [ 7281.633649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7281.634813] LustreError: 435039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.544919] Lustre: server umount lustre-OST0000 complete [ 7290.628172] Lustre: server umount lustre-MDT0000 complete [ 7295.643287] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7297.060712] Key type lgssc unregistered [ 7297.225609] LNet: 439792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7297.230462] LNetError: 439792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7297.242583] LNet: Removed LNI 192.168.201.103@tcp [ 7297.675157] Key type .llcrypt unregistered [ 7297.676227] Key type ._llcrypt unregistered [ 7309.564119] Key type ._llcrypt registered [ 7309.565169] Key type .llcrypt registered [ 7309.603837] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7309.729277] Key type .llcrypt unregistered [ 7309.730620] Key type ._llcrypt unregistered [ 7315.883364] Key type ._llcrypt registered [ 7315.884530] Key type .llcrypt registered [ 7315.919548] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7316.247868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.269027] alg: No test for adler32 (adler32-zlib) [ 7317.110700] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 7317.190205] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7318.767183] Key type lgssc registered [ 7319.263700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7329.427958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7329.433938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.514628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7330.523808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7330.558747] Lustre: lustre-MDT0000: new disk, initializing [ 7330.579105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.584449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7331.889454] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7337.119645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.194950] Lustre: 443155:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7337.248476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7337.255213] Lustre: Skipped 1 previous similar message [ 7337.313256] Lustre: lustre-MDT0001: new disk, initializing [ 7337.374748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7337.398930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7337.405639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7339.329440] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7343.769262] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7346.012325] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7349.104225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7349.215695] Lustre: lustre-OST0000: new disk, initializing [ 7349.218514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7349.222652] Lustre: 445122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7349.259466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7351.483064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7351.486037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7351.521253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7351.752060] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7356.089128] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7362.018429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.021054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.023463] Lustre: Skipped 2 previous similar messages [ 7362.026082] Lustre: Skipped 1 previous similar message [ 7364.653351] Lustre: server umount lustre-MDT0000 complete [ 7366.091816] LustreError: 445121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782537430 with bad export cookie 13723125597254021017 [ 7366.095606] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7366.096498] LustreError: 445121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7366.231273] Lustre: server umount lustre-MDT0001 complete [ 7377.960183] Lustre: server umount lustre-OST0000 complete [ 7382.080218] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:17:25 (1782537445) [ 7384.886924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.085179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.426821] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7389.371291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7389.556341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.090439] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7393.526892] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7395.645450] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7411.684864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.813257] Lustre: Found index 0 for lustre-OST0000, updating log [ 7411.897175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.450171] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7417.090259] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7431.334502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7435.744250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7435.747212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.751922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.769248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.771905] Lustre: Skipped 2 previous similar messages [ 7440.076803] Lustre: server umount lustre-OST0000 complete [ 7453.152564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.153100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.155743] Lustre: Skipped 2 previous similar messages [ 7453.158257] Lustre: Skipped 1 previous similar message [ 7458.271926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7458.274435] Lustre: Skipped 1 previous similar message [ 7458.576064] Lustre: server umount lustre-MDT0000 complete [ 7459.660969] LustreError: 449660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782537523 with bad export cookie 13723125597254022410 [ 7459.663532] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7459.663873] LustreError: 449660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7459.768132] Lustre: server umount lustre-MDT0001 complete [ 7462.268484] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:18:46 (1782537526) [ 7479.007191] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [ 8045.446604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.611708] LustreError: 453570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.618430] LustreError: 453570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8045.639496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.009811] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8049.864642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.002440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8051.304679] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8054.335272] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8056.409611] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8071.970665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8072.064365] Lustre: Found index 0 for lustre-OST0000, updating log [ 8072.169638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.197423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 8074.536517] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8077.708130] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8081.378945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.383650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.392130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.395448] Lustre: Skipped 1 previous similar message [ 8084.683420] Lustre: server umount lustre-OST0000 complete [ 8085.984344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8085.987337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8085.991881] Lustre: Skipped 1 previous similar message [ 8085.994098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8087.010283] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.033401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8088.037460] Lustre: Skipped 3 previous similar messages [ 8092.129385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.131569] Lustre: Skipped 7 previous similar messages [ 8100.319193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8100.321097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8100.329464] Lustre: Skipped 15 previous similar messages [ 8100.408529] Lustre: server umount lustre-MDT0000 complete [ 8101.344741] LustreError: 453570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.351624] LustreError: 453570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8102.105418] LustreError: 455472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538166 with bad export cookie 13723125597254023481 [ 8102.109958] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8102.111670] LustreError: 455472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8102.229693] Lustre: server umount lustre-MDT0001 complete [ 8106.719369] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [ 8121.652020] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8125.564948] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8141.412450] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:30:04 (1782538204) [ 8149.009942] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8150.435512] Key type lgssc unregistered [ 8150.569319] LNet: 460229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8150.571565] LNetError: 460229:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8201.765419] LNet: Removed LNI 192.168.201.103@tcp [ 8202.227138] Key type .llcrypt unregistered [ 8202.229206] Key type ._llcrypt unregistered [ 8210.314439] Key type ._llcrypt registered [ 8210.316044] Key type .llcrypt registered [ 8210.373644] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8210.671943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8210.714487] alg: No test for adler32 (adler32-zlib) [ 8211.561193] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8211.648813] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8213.231194] Key type lgssc registered [ 8213.735841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8215.561916] LNet: Added LNI 192.168.201.103@tcp100 [8/256/0/180] [ 8216.000696] LNet: Added LNI 192.168.201.103@tcp101 [8/256/0/180] [ 8216.437372] LNet: Added LNI 192.168.201.103@tcp102 [8/256/0/180] [ 8216.873822] LNet: Added LNI 192.168.201.103@tcp103 [8/256/0/180] [ 8217.293430] LNet: Added LNI 192.168.201.103@tcp104 [8/256/0/180] [ 8217.715918] LNet: Added LNI 192.168.201.103@tcp105 [8/256/0/180] [ 8218.152606] LNet: Added LNI 192.168.201.103@tcp106 [8/256/0/180] [ 8218.594344] LNet: Added LNI 192.168.201.103@tcp107 [8/256/0/180] [ 8219.039753] LNet: Added LNI 192.168.201.103@tcp108 [8/256/0/180] [ 8219.477548] LNet: Added LNI 192.168.201.103@tcp109 [8/256/0/180] [ 8219.910917] LNet: Added LNI 192.168.201.103@tcp110 [8/256/0/180] [ 8220.350621] LNet: Added LNI 192.168.201.103@tcp111 [8/256/0/180] [ 8220.791571] LNet: Added LNI 192.168.201.103@tcp112 [8/256/0/180] [ 8221.227223] LNet: Added LNI 192.168.201.103@tcp113 [8/256/0/180] [ 8221.675476] LNet: Added LNI 192.168.201.103@tcp114 [8/256/0/180] [ 8222.101867] LNet: Added LNI 192.168.201.103@tcp115 [8/256/0/180] [ 8222.525171] LNet: Added LNI 192.168.201.103@tcp116 [8/256/0/180] [ 8222.948627] LNet: Added LNI 192.168.201.103@tcp117 [8/256/0/180] [ 8223.306376] LNet: Added LNI 192.168.201.103@tcp118 [8/256/0/180] [ 8223.685059] LNet: Added LNI 192.168.201.103@tcp119 [8/256/0/180] [ 8224.099122] LNet: Added LNI 192.168.201.103@tcp120 [8/256/0/180] [ 8224.526882] LNet: Added LNI 192.168.201.103@tcp121 [8/256/0/180] [ 8224.968793] LNet: Added LNI 192.168.201.103@tcp122 [8/256/0/180] [ 8225.381246] LNet: Added LNI 192.168.201.103@tcp123 [8/256/0/180] [ 8225.788424] LNet: Added LNI 192.168.201.103@tcp124 [8/256/0/180] [ 8226.181982] LNet: Added LNI 192.168.201.103@tcp125 [8/256/0/180] [ 8226.599198] LNet: Added LNI 192.168.201.103@tcp126 [8/256/0/180] [ 8227.034345] LNet: Added LNI 192.168.201.103@tcp127 [8/256/0/180] [ 8227.458498] LNet: Added LNI 192.168.201.103@tcp128 [8/256/0/180] [ 8227.894726] LNet: Added LNI 192.168.201.103@tcp129 [8/256/0/180] [ 8228.328483] LNet: Added LNI 192.168.201.103@tcp130 [8/256/0/180] [ 8228.750036] LNet: Added LNI 192.168.201.103@tcp131 [8/256/0/180] [ 8229.190811] LNet: Added LNI 192.168.201.103@tcp132 [8/256/0/180] [ 8229.618766] LNet: Added LNI 192.168.201.103@tcp133 [8/256/0/180] [ 8230.044801] LNet: Added LNI 192.168.201.103@tcp134 [8/256/0/180] [ 8230.462867] LNet: Added LNI 192.168.201.103@tcp135 [8/256/0/180] [ 8233.644801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8233.654487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.710473] Lustre: 463241:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 8234.742674] LNetError: 460703:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8234.745510] LNetError: Protocol error connecting to 192.168.201.103@tcp116 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8234.748290] LNetError: Skipped 1 previous similar message [ 8234.791967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8234.800999] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8234.837994] Lustre: lustre-MDT0000: new disk, initializing [ 8234.858701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.864288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8235.231406] LNet: 5 local NIs in recovery (showing 5): 192.168.201.103@tcp122, 192.168.201.103@tcp123, 192.168.201.103@tcp125, 192.168.201.103@tcp126, 192.168.201.103@tcp127 [ 8236.648586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8237.285118] LNetError: 463269:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8237.291150] LNetError: 463269:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 19 previous similar messages [ 8237.296464] LNetError: Protocol error connecting to 192.168.201.103@tcp123 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8237.302795] LNetError: Skipped 21 previous similar messages [ 8241.377262] LNetError: 463261:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8241.383714] LNetError: 463261:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 8241.388434] LNetError: Protocol error connecting to 192.168.201.103@tcp123 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8242.269615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.348678] Lustre: 463256:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8242.393505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8242.397016] Lustre: Skipped 1 previous similar message [ 8242.446782] Lustre: lustre-MDT0001: new disk, initializing [ 8242.460294] Lustre: 464060:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 8242.464390] Lustre: 464060:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8242.527652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8242.548174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8242.554927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8244.327312] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8244.671324] Lustre: 464351:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 8244.677101] Lustre: 464351:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8248.414727] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8249.568619] LNetError: 463274:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8249.574276] LNetError: Protocol error connecting to 192.168.201.103@tcp123 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8250.934662] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8254.183918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8254.342933] Lustre: lustre-OST0000: new disk, initializing [ 8254.345380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8254.348947] Lustre: 465232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8254.401803] Lustre: 465237:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 8254.407658] Lustre: 465237:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8254.434936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.940805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8261.543281] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8263.679538] Lustre: 466058:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [ 8263.685302] Lustre: 466058:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8263.715376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8263.724755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8263.786305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8264.928622] LNetError: 460705:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8264.935134] LNetError: Protocol error connecting to 192.168.201.103@tcp116 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8268.772387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.772579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.773864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.787586] Lustre: Skipped 1 previous similar message [ 8270.890060] Lustre: server umount lustre-OST0000 complete [ 8273.120732] LNetError: 460704:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [ 8273.126646] LNetError: 460704:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 8273.131346] LNetError: Protocol error connecting to 192.168.201.103@tcp128 at host 192.168.201.103:988: is it running a compatible version of Lustre? [ 8273.137466] LNetError: Skipped 8 previous similar messages [ 8273.376248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8273.379444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8273.386067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8273.390227] Lustre: Skipped 1 previous similar message [ 8278.641094] Lustre: server umount lustre-MDT0000 complete [ 8280.395081] LustreError: 464059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538344 with bad export cookie 9717561151778004107 [ 8280.402039] LustreError: 464059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8280.402279] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8280.556667] Lustre: server umount lustre-MDT0001 complete [ 8285.185207] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8286.621523] Key type lgssc unregistered [ 8286.832247] LNet: 467146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8286.837756] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8286.842183] LNet: Removed LNI 192.168.201.103@tcp [ 8287.903177] LNet: Removed LNI 192.168.201.103@tcp100 [ 8287.905778] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8287.910121] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8287.914873] LNet: Removed LNI 192.168.201.103@tcp101 [ 8288.927179] LNet: Removed LNI 192.168.201.103@tcp102 [ 8288.929736] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8288.934085] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8288.938847] LNet: Removed LNI 192.168.201.103@tcp103 [ 8289.951231] LNet: Removed LNI 192.168.201.103@tcp104 [ 8290.975204] LNet: Removed LNI 192.168.201.103@tcp105 [ 8290.977795] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8290.982161] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8290.986929] LNet: Removed LNI 192.168.201.103@tcp106 [ 8291.999129] LNet: Removed LNI 192.168.201.103@tcp107 [ 8293.023204] LNet: Removed LNI 192.168.201.103@tcp108 [ 8294.047181] LNet: Removed LNI 192.168.201.103@tcp109 [ 8295.071212] LNet: Removed LNI 192.168.201.103@tcp110 [ 8295.073858] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8295.077843] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 8295.082198] LNet: Removed LNI 192.168.201.103@tcp111 [ 8296.095212] LNet: Removed LNI 192.168.201.103@tcp112 [ 8297.119176] LNet: Removed LNI 192.168.201.103@tcp113 [ 8298.143203] LNet: Removed LNI 192.168.201.103@tcp114 [ 8299.167310] LNet: Removed LNI 192.168.201.103@tcp115 [ 8300.191179] LNet: Removed LNI 192.168.201.103@tcp116 [ 8301.215253] LNet: Removed LNI 192.168.201.103@tcp117 [ 8302.239274] LNet: Removed LNI 192.168.201.103@tcp118 [ 8303.263224] LNet: Removed LNI 192.168.201.103@tcp119 [ 8303.265907] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8303.270313] LNetError: 467146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 8303.275161] LNet: Removed LNI 192.168.201.103@tcp120 [ 8304.287221] LNet: Removed LNI 192.168.201.103@tcp121 [ 8305.311200] LNet: Removed LNI 192.168.201.103@tcp122 [ 8306.335195] LNet: Removed LNI 192.168.201.103@tcp123 [ 8307.359240] LNet: Removed LNI 192.168.201.103@tcp124 [ 8308.383194] LNet: Removed LNI 192.168.201.103@tcp125 [ 8309.407189] LNet: Removed LNI 192.168.201.103@tcp126 [ 8310.431205] LNet: Removed LNI 192.168.201.103@tcp127 [ 8311.455192] LNet: Removed LNI 192.168.201.103@tcp128 [ 8312.479155] LNet: Removed LNI 192.168.201.103@tcp129 [ 8312.481731] LNet: Removed LNI 192.168.201.103@tcp130 [ 8312.484091] LNet: Removed LNI 192.168.201.103@tcp131 [ 8312.486487] LNet: Removed LNI 192.168.201.103@tcp132 [ 8312.488834] LNet: Removed LNI 192.168.201.103@tcp133 [ 8312.491173] LNet: Removed LNI 192.168.201.103@tcp134 [ 8312.499871] LNet: Removed LNI 192.168.201.103@tcp135 [ 8312.907878] Key type .llcrypt unregistered [ 8312.909714] Key type ._llcrypt unregistered [ 8323.194487] Key type ._llcrypt registered [ 8323.196977] Key type .llcrypt registered [ 8323.251249] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8330.396582] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8330.748521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8330.797902] alg: No test for adler32 (adler32-zlib) [ 8331.644240] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8331.734712] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8333.311173] Key type lgssc registered [ 8333.836923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8350.011099] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:33:33 (1782538413) [ 8357.761291] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8359.063098] Key type lgssc unregistered [ 8359.198522] LNet: 471206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8359.202695] LNetError: 471206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8359.213516] LNet: Removed LNI 192.168.201.103@tcp [ 8359.609445] Key type .llcrypt unregistered [ 8359.610982] Key type ._llcrypt unregistered [ 8367.319203] Key type ._llcrypt registered [ 8367.320403] Key type .llcrypt registered [ 8367.377607] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8367.713510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8367.741080] alg: No test for adler32 (adler32-zlib) [ 8368.583748] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8368.677907] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8370.263171] Key type lgssc registered [ 8370.856453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8374.595364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8374.599779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.675986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8375.684606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8375.717847] Lustre: lustre-MDT0000: new disk, initializing [ 8375.736807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.742071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8377.237779] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8382.475798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.538072] Lustre: 472394:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8382.571226] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8382.574832] Lustre: Skipped 1 previous similar message [ 8382.620697] Lustre: lustre-MDT0001: new disk, initializing [ 8382.667437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.686910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8382.693334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8384.382668] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8388.964212] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8391.409193] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8394.030144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8394.203943] Lustre: lustre-OST0000: new disk, initializing [ 8394.207132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8394.211824] Lustre: 474365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8394.261850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8396.986169] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8401.188169] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8402.602679] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lnet_if_list [ 8403.977898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8403.981923] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8404.021489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8404.464080] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8404.479615] LNet: Added LNI 192.168.201.103@tcp10 [8/256/0/180] [ 8406.555877] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8406.569932] LNet: Added LNI 192.168.201.103@tcp11 [8/256/0/180] [ 8408.124846] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8408.139889] LNet: Added LNI 192.168.201.103@tcp12 [8/256/0/180] [ 8409.828651] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8409.840578] LNet: Added LNI 192.168.201.103@tcp13 [8/256/0/180] [ 8411.706695] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8411.721688] LNet: Added LNI 192.168.201.103@tcp14 [8/256/0/180] [ 8413.463548] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8413.480694] LNet: Added LNI 192.168.201.103@tcp15 [8/256/0/180] [ 8415.355571] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8415.371287] LNet: Added LNI 192.168.201.103@tcp16 [8/256/0/180] [ 8417.097347] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8417.113249] LNet: Added LNI 192.168.201.103@tcp17 [8/256/0/180] [ 8419.065796] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8419.083788] LNet: Added LNI 192.168.201.103@tcp18 [8/256/0/180] [ 8421.099715] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8421.111692] LNet: Added LNI 192.168.201.103@tcp19 [8/256/0/180] [ 8423.233239] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8423.252097] LNet: Added LNI 192.168.201.103@tcp20 [8/256/0/180] [ 8425.519793] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8425.538759] LNet: Added LNI 192.168.201.103@tcp21 [8/256/0/180] [ 8428.006265] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8428.023938] LNet: Added LNI 192.168.201.103@tcp22 [8/256/0/180] [ 8430.428358] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8430.445977] LNet: Added LNI 192.168.201.103@tcp23 [8/256/0/180] [ 8432.830785] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8432.841411] LNet: Added LNI 192.168.201.103@tcp24 [8/256/0/180] [ 8435.062898] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8435.076548] LNet: Added LNI 192.168.201.103@tcp25 [8/256/0/180] [ 8437.615497] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8437.633293] LNet: Added LNI 192.168.201.103@tcp26 [8/256/0/180] [ 8440.279603] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8440.298210] LNet: Added LNI 192.168.201.103@tcp27 [8/256/0/180] [ 8442.387063] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8442.403662] LNet: Added LNI 192.168.201.103@tcp28 [8/256/0/180] [ 8444.812576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8444.826468] LNet: Added LNI 192.168.201.103@tcp29 [8/256/0/180] [ 8447.345431] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8447.359161] LNet: Added LNI 192.168.201.103@tcp30 [8/256/0/180] [ 8449.519263] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8449.527834] LNet: Added LNI 192.168.201.103@tcp31 [8/256/0/180] [ 8451.952876] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8451.961606] LNet: Added LNI 192.168.201.103@tcp32 [8/256/0/180] [ 8453.845289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8453.858098] LNet: Added LNI 192.168.201.103@tcp33 [8/256/0/180] [ 8456.368850] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8456.384551] LNet: Added LNI 192.168.201.103@tcp34 [8/256/0/180] [ 8459.110218] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8459.124748] LNet: Added LNI 192.168.201.103@tcp35 [8/256/0/180] [ 8461.964782] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8461.982984] LNet: Added LNI 192.168.201.103@tcp36 [8/256/0/180] [ 8464.792078] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8464.810669] LNet: Added LNI 192.168.201.103@tcp37 [8/256/0/180] [ 8467.664720] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8467.676735] LNet: Added LNI 192.168.201.103@tcp38 [8/256/0/180] [ 8470.644643] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8470.664440] LNet: Added LNI 192.168.201.103@tcp39 [8/256/0/180] [ 8473.971797] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8473.991838] LNet: Added LNI 192.168.201.103@tcp40 [8/256/0/180] [ 8477.285000] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8477.299060] LNet: Added LNI 192.168.201.103@tcp41 [8/256/0/180] [ 8480.726284] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8480.742594] LNet: Added LNI 192.168.201.103@tcp42 [8/256/0/180] [ 8484.081343] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8484.096543] LNet: Added LNI 192.168.201.103@tcp43 [8/256/0/180] [ 8487.540057] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8487.556314] LNet: Added LNI 192.168.201.103@tcp44 [8/256/0/180] [ 8491.112180] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8491.131786] LNet: Added LNI 192.168.201.103@tcp45 [8/256/0/180] [ 8494.583261] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8494.601689] LNet: Added LNI 192.168.201.103@tcp46 [8/256/0/180] [ 8497.934636] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8497.945455] LNet: Added LNI 192.168.201.103@tcp47 [8/256/0/180] [ 8501.376167] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8501.390472] LNet: Added LNI 192.168.201.103@tcp48 [8/256/0/180] [ 8504.926363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8504.944266] LNet: Added LNI 192.168.201.103@tcp49 [8/256/0/180] [ 8508.317575] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8508.334051] LNet: Added LNI 192.168.201.103@tcp50 [8/256/0/180] [ 8521.184733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8521.189529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.198136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.701075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.702320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8521.709238] Lustre: Skipped 1 previous similar message [ 8521.714894] Lustre: Skipped 1 previous similar message [ 8523.323200] Lustre: server umount lustre-MDT0000 complete [ 8525.120045] LustreError: 472388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538589 with bad export cookie 14235394864976258744 [ 8525.125918] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8525.127197] LustreError: 472388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8525.292256] Lustre: server umount lustre-MDT0001 complete [ 8534.632952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8538.286783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.111155] Lustre: 471668:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782538590/real 1782538590] req@000000005facee62 x1869127018964096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782538606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8542.126976] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8546.207143] Lustre: 471668:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782538594/real 1782538594] req@0000000008391706 x1869127018964224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782538610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8547.295235] Lustre: 471670:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782538595/real 1782538595] req@0000000082c4fc64 x1869127018964352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782538611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8550.367540] LustreError: 471667:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c844f34c x1869127018965120/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8550.550057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.391141] Lustre: 471671:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782538599/real 1782538599] req@0000000080f3202d x1869127018964608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782538615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8552.220678] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8556.343224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8556.536320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8557.535163] Lustre: 471668:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782538605/real 1782538605] req@000000003394f848 x1869127018964864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782538621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8558.259130] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8561.545764] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8561.642629] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8564.064348] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8576.480515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8576.485066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8576.493204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8579.823506] Lustre: server umount lustre-OST0000 complete [ 8582.114539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8582.114789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.118904] LustreError: Skipped 1 previous similar message [ 8582.125490] Lustre: Skipped 1 previous similar message [ 8582.126623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8582.133352] Lustre: Skipped 2 previous similar messages [ 8587.233360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8587.237235] Lustre: Skipped 1 previous similar message [ 8587.824992] Lustre: server umount lustre-MDT0000 complete [ 8589.634926] LustreError: 472431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538653 with bad export cookie 14235394864976260844 [ 8589.640780] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8589.642051] LustreError: 472431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8589.799457] Lustre: server umount lustre-MDT0001 complete [ 8594.799790] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8596.258852] Key type lgssc unregistered [ 8596.492319] LNet: 501330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8596.499852] LNetError: 501330:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8596.504960] LNet: Removed LNI 192.168.201.103@tcp [ 8596.507560] LNet: Removed LNI 192.168.201.103@tcp10 [ 8596.510234] LNet: Removed LNI 192.168.201.103@tcp11 [ 8596.513055] LNet: Removed LNI 192.168.201.103@tcp12 [ 8596.516230] LNet: Removed LNI 192.168.201.103@tcp13 [ 8596.518858] LNet: Removed LNI 192.168.201.103@tcp14 [ 8596.521429] LNet: Removed LNI 192.168.201.103@tcp15 [ 8596.523813] LNet: Removed LNI 192.168.201.103@tcp16 [ 8596.526473] LNet: Removed LNI 192.168.201.103@tcp17 [ 8596.528854] LNet: Removed LNI 192.168.201.103@tcp18 [ 8596.531259] LNet: Removed LNI 192.168.201.103@tcp19 [ 8596.533611] LNet: Removed LNI 192.168.201.103@tcp20 [ 8596.535841] LNet: Removed LNI 192.168.201.103@tcp21 [ 8596.538216] LNet: Removed LNI 192.168.201.103@tcp22 [ 8596.540466] LNet: Removed LNI 192.168.201.103@tcp23 [ 8596.542593] LNet: Removed LNI 192.168.201.103@tcp24 [ 8596.544717] LNet: Removed LNI 192.168.201.103@tcp25 [ 8596.546806] LNet: Removed LNI 192.168.201.103@tcp26 [ 8596.548825] LNet: Removed LNI 192.168.201.103@tcp27 [ 8596.550762] LNet: Removed LNI 192.168.201.103@tcp28 [ 8596.552885] LNet: Removed LNI 192.168.201.103@tcp29 [ 8596.554955] LNet: Removed LNI 192.168.201.103@tcp30 [ 8596.556900] LNet: Removed LNI 192.168.201.103@tcp31 [ 8596.558793] LNet: Removed LNI 192.168.201.103@tcp32 [ 8596.560746] LNet: Removed LNI 192.168.201.103@tcp33 [ 8596.562755] LNet: Removed LNI 192.168.201.103@tcp34 [ 8596.564639] LNet: Removed LNI 192.168.201.103@tcp35 [ 8596.566464] LNet: Removed LNI 192.168.201.103@tcp36 [ 8596.568188] LNet: Removed LNI 192.168.201.103@tcp37 [ 8596.570119] LNet: Removed LNI 192.168.201.103@tcp38 [ 8596.571822] LNet: Removed LNI 192.168.201.103@tcp39 [ 8596.573516] LNet: Removed LNI 192.168.201.103@tcp40 [ 8596.575125] LNet: Removed LNI 192.168.201.103@tcp41 [ 8596.576717] LNet: Removed LNI 192.168.201.103@tcp42 [ 8596.578270] LNet: Removed LNI 192.168.201.103@tcp43 [ 8596.579788] LNet: Removed LNI 192.168.201.103@tcp44 [ 8596.581320] LNet: Removed LNI 192.168.201.103@tcp45 [ 8596.582776] LNet: Removed LNI 192.168.201.103@tcp46 [ 8596.584307] LNet: Removed LNI 192.168.201.103@tcp47 [ 8596.585764] LNet: Removed LNI 192.168.201.103@tcp48 [ 8596.587345] LNet: Removed LNI 192.168.201.103@tcp49 [ 8596.593158] LNet: Removed LNI 192.168.201.103@tcp50 [ 8597.036972] Key type .llcrypt unregistered [ 8597.039189] Key type ._llcrypt unregistered [ 8606.189922] Key type ._llcrypt registered [ 8606.190959] Key type .llcrypt registered [ 8606.240328] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8614.310214] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8614.681412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8614.741259] alg: No test for adler32 (adler32-zlib) [ 8615.654640] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8615.793577] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8617.383185] Key type lgssc registered [ 8618.053632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8634.441990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8634.452794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.559623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8635.572233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8635.614591] Lustre: lustre-MDT0000: new disk, initializing [ 8635.643516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.655558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8637.493227] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8643.203791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.274780] Lustre: 504502:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8643.319543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8643.322907] Lustre: Skipped 1 previous similar message [ 8643.368066] Lustre: lustre-MDT0001: new disk, initializing [ 8643.417483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8643.436645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8643.441448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8645.444771] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8650.425525] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8653.048152] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8656.891054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8657.011373] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8657.047428] Lustre: lustre-OST0000: new disk, initializing [ 8657.050401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8657.054747] Lustre: 506518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8657.098297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.736870] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8660.664160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8660.666994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8660.705441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8664.636145] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8671.205578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8671.206112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.207802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.211367] LustreError: Skipped 1 previous similar message [ 8671.219337] Lustre: Skipped 1 previous similar message [ 8673.646175] Lustre: server umount lustre-OST0000 complete [ 8676.322526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.323843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.333822] Lustre: Skipped 1 previous similar message [ 8676.338041] Lustre: Skipped 1 previous similar message [ 8681.441031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8681.445170] Lustre: Skipped 1 previous similar message [ 8681.719860] Lustre: server umount lustre-MDT0000 complete [ 8683.522082] LustreError: 504494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538747 with bad export cookie 17610660645234061383 [ 8683.525557] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8683.526214] LustreError: 504494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8683.668191] Lustre: server umount lustre-MDT0001 complete [ 8688.654447] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8690.049381] Key type lgssc unregistered [ 8690.211474] LNet: 508421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8690.213385] LNetError: 508421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8690.222272] LNet: Removed LNI 192.168.201.103@tcp [ 8690.624476] Key type .llcrypt unregistered [ 8690.626057] Key type ._llcrypt unregistered [ 8698.594050] Key type ._llcrypt registered [ 8698.595100] Key type .llcrypt registered [ 8698.645721] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8698.981452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8699.035251] alg: No test for adler32 (adler32-zlib) [ 8699.894233] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8699.985807] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8701.559222] Key type lgssc registered [ 8702.103981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8705.736967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8705.748798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8706.931086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8708.874232] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8713.080545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.209136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8714.571516] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8717.727237] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8719.524080] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8721.936140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8722.039172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.884786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8726.229489] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8736.386052] loop0: detected capacity change from 0 to 204800000 [ 8736.509148] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8736.515574] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8736.523212] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8739.575145] loop0: detected capacity change from 0 to 204800000 [ 8739.647843] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8739.651429] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8739.655525] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8742.128550] loop0: detected capacity change from 0 to 204800000 [ 8744.695591] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8744.772898] Lustre: fs15246-OST0000: new disk, initializing [ 8744.776138] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8744.780954] Lustre: 513262:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8744.812235] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8747.442609] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8751.888480] loop1: detected capacity change from 0 to 204800000 [ 8753.877354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.910262] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8753.929792] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8753.965880] Lustre: fs15246-MDT0000: new disk, initializing [ 8754.015336] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.023065] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8754.060981] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8755.103663] Lustre: 514382:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000005a659b7e x1869127365986560/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782538835 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8755.205760] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8764.196536] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8764.202637] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8764.216960] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8768.998664] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.007702] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.363360] Lustre: server umount fs15246-MDT0000 complete [ 8774.118062] LustreError: 514387:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.125392] LustreError: 514387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8775.542471] Lustre: server umount fs15246-OST0000 complete [ 8782.816097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8782.817911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.821107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.352276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.354190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.065919] Lustre: server umount lustre-OST0000 complete [ 8789.474198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8789.475446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.481780] Lustre: Skipped 1 previous similar message [ 8789.487393] Lustre: Skipped 1 previous similar message [ 8793.313440] Lustre: server umount lustre-MDT0000 complete [ 8794.592954] LustreError: 509667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8794.602731] LustreError: 509667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8795.032804] LustreError: 515522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538859 with bad export cookie 11336319082142416720 [ 8795.038873] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.039659] LustreError: 515522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8795.197103] Lustre: server umount lustre-MDT0001 complete [ 8799.943203] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8801.346113] Key type lgssc unregistered [ 8801.497720] LNet: 517478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8801.502303] LNetError: 517478:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8801.510552] LNet: Removed LNI 192.168.201.103@tcp [ 8801.978248] Key type .llcrypt unregistered [ 8801.980358] Key type ._llcrypt unregistered [ 8811.677299] Key type ._llcrypt registered [ 8811.678332] Key type .llcrypt registered [ 8811.715361] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8812.070606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8812.110676] alg: No test for adler32 (adler32-zlib) [ 8812.968261] Lustre: Lustre: Build Version: 2.17.54_99_gde77dd7 [ 8813.066933] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8814.655240] Key type lgssc registered [ 8815.104977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8833.021807] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8837.354057] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8854.359349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8854.369217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.464643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8855.475564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8855.514138] Lustre: lustre-MDT0000: new disk, initializing [ 8855.537192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8855.542735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8857.239434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8862.789513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.855783] Lustre: 521912:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8862.899706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8862.903434] Lustre: Skipped 1 previous similar message [ 8862.951380] Lustre: lustre-MDT0001: new disk, initializing [ 8862.985232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8863.002562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8863.006683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8864.568868] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8869.553454] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8872.274990] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8875.124475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.283286] Lustre: lustre-OST0000: new disk, initializing [ 8875.286590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8875.291697] Lustre: 523881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8875.344146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8877.762913] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8880.135487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8880.145133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8880.188834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8882.530857] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8884.294307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8884.378781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8886.201767] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8886.303290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8890.340381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8890.340953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.342474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.343803] LustreError: Skipped 1 previous similar message [ 8890.349603] Lustre: Skipped 1 previous similar message [ 8893.154636] Lustre: server umount lustre-OST0000 complete [ 8895.456940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8895.457441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8895.460489] Lustre: Skipped 1 previous similar message [ 8895.463405] Lustre: Skipped 2 previous similar messages [ 8900.576907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8900.580415] Lustre: Skipped 1 previous similar message [ 8900.721283] Lustre: server umount lustre-MDT0000 complete [ 8902.400117] LustreError: 521905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782538966 with bad export cookie 17614688029156930438 [ 8902.406779] LustreError: 521905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8902.406889] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8902.564337] Lustre: server umount lustre-MDT0001 complete [ 8913.717569] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8918.491100] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8933.773250] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8620 sec ========== 01:43:17 (1782538997) [ 8934.464703] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:43:18 (1782538998) === [ 8936.078760] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:43:19 (1782538999) === [ 8954.383875] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8955.991941] Key type lgssc unregistered [ 8956.167570] LNet: 530515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8956.172494] LNetError: 530515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8956.181795] LNet: Removed LNI 192.168.201.103@tcp [ 8956.674552] Key type .llcrypt unregistered [ 8956.676514] Key type ._llcrypt unregistered