[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487984292 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003288] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007958] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.021535] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026101] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028133] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.030428] *** VALIDATE proc *** [ 0.032061] *** VALIDATE cgroup *** [ 0.033000] *** VALIDATE cgroup2 *** [ 0.034225] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036026] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042231] debug: unmapping init [mem 0xffffffffa0459000-0xffffffffa0460fff] [ 0.044999] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.054235] rcu: Hierarchical SRCU implementation. [ 0.056572] smp: Bringing up secondary CPUs ... [ 0.057596] x86: Booting SMP configuration: [ 0.058019] .... node #0, CPUs: #1 #2 #3 [ 0.063028] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.233418] node 0 deferred pages initialised in 166ms [ 0.237530] devtmpfs: initialized [ 0.238242] x86/mm: Memory block size: 128MB [ 0.240920] gcov: version magic: 0x41383552 [ 0.242181] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244258] pinctrl core: initialized pinctrl subsystem [ 0.245200] [ 0.245692] ************************************************************* [ 0.246029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247023] ** ** [ 0.248019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249016] ** ** [ 0.250017] ** This means that this kernel is built to expose internal ** [ 0.251018] ** IOMMU data structures, which may compromise security on ** [ 0.252016] ** your system. ** [ 0.253015] ** ** [ 0.254017] ** If you see this message and you are not debugging the ** [ 0.255019] ** kernel, report this immediately to your vendor! ** [ 0.256016] ** ** [ 0.257017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258022] ************************************************************* [ 0.260193] NET: Registered protocol family 16 [ 0.261614] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.262064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.264465] cpuidle: using governor menu [ 0.269465] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.271632] PCI: Using configuration type 1 for base access [ 0.274145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286289] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.287016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288314] cryptd: max_cpu_qlen set to 1000 [ 0.289274] ACPI: Added _OSI(Module Device) [ 0.290019] ACPI: Added _OSI(Processor Device) [ 0.291014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.292015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296237] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.299192] ACPI: Interpreter enabled [ 0.300066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301018] ACPI: Using IOAPIC for interrupt routing [ 0.302116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303922] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.315041] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.318066] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.323085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.327829] acpiphp: Slot [2] registered [ 0.329172] acpiphp: Slot [5] registered [ 0.331154] acpiphp: Slot [6] registered [ 0.332129] acpiphp: Slot [7] registered [ 0.334129] acpiphp: Slot [8] registered [ 0.335119] acpiphp: Slot [9] registered [ 0.337141] acpiphp: Slot [10] registered [ 0.339136] acpiphp: Slot [3] registered [ 0.340178] acpiphp: Slot [4] registered [ 0.342139] acpiphp: Slot [11] registered [ 0.344160] acpiphp: Slot [12] registered [ 0.346199] acpiphp: Slot [13] registered [ 0.347145] acpiphp: Slot [14] registered [ 0.349146] acpiphp: Slot [15] registered [ 0.351124] acpiphp: Slot [16] registered [ 0.352135] acpiphp: Slot [17] registered [ 0.354146] acpiphp: Slot [18] registered [ 0.355148] acpiphp: Slot [19] registered [ 0.357154] acpiphp: Slot [20] registered [ 0.358142] acpiphp: Slot [21] registered [ 0.360149] acpiphp: Slot [22] registered [ 0.362148] acpiphp: Slot [23] registered [ 0.363142] acpiphp: Slot [24] registered [ 0.365174] acpiphp: Slot [25] registered [ 0.367158] acpiphp: Slot [26] registered [ 0.368147] acpiphp: Slot [27] registered [ 0.369239] acpiphp: Slot [28] registered [ 0.370095] acpiphp: Slot [29] registered [ 0.372110] acpiphp: Slot [30] registered [ 0.373137] acpiphp: Slot [31] registered [ 0.374132] PCI host bridge to bus 0000:00 [ 0.376040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.379047] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.385035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.388033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.391045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.393189] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.396143] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.399364] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.413594] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.416025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.419027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.422026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.426685] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.429891] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.432054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.435987] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.442020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.454018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.460025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.467547] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.476020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.482022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510190] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.516021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.525021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.544024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.554177] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.563025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.569021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.588024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612034] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.619021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.634018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.644982] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.658017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.664018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.687473] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.697024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.706025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.729023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.743733] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.746473] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.748408] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.751425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.753321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.760138] iommu: Default domain type: Passthrough [ 0.762564] SCSI subsystem initialized [ 0.764219] ACPI: bus type USB registered [ 0.766121] usbcore: registered new interface driver usbfs [ 0.768124] usbcore: registered new interface driver hub [ 0.770153] usbcore: registered new device driver usb [ 0.772220] pps_core: LinuxPPS API ver. 1 registered [ 0.774016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.777100] PTP clock support registered [ 0.780173] EDAC MC: Ver: 3.0.0 [ 0.782212] PCI: Using ACPI for IRQ routing [ 0.783756] NetLabel: Initializing [ 0.785018] NetLabel: domain hash size = 128 [ 0.787023] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.789182] NetLabel: unlabeled traffic allowed by default [ 0.792159] vgaarb: loaded [ 0.794368] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.796087] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.805190] clocksource: Switched to clocksource kvm-clock [ 0.919914] VFS: Disk quotas dquot_6.6.0 [ 0.922115] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.925145] *** VALIDATE ramfs *** [ 0.926744] *** VALIDATE hugetlbfs *** [ 0.929201] pnp: PnP ACPI init [ 0.931857] pnp: PnP ACPI: found 6 devices [ 0.960219] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.964035] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.966249] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.968544] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.971390] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.973708] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.976608] NET: Registered protocol family 2 [ 0.979886] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.986601] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.991461] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.997840] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.002209] TCP: Hash tables configured (established 65536 bind 65536) [ 1.005587] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.009818] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013304] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016700] NET: Registered protocol family 1 [ 1.022427] RPC: Registered named UNIX socket transport module. [ 1.024759] RPC: Registered udp transport module. [ 1.026304] RPC: Registered tcp transport module. [ 1.027650] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.029290] NET: Registered protocol family 44 [ 1.031217] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.034072] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.036917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.040202] PCI: CLS 0 bytes, default 64 [ 1.042161] Unpacking initramfs... [ 2.479601] debug: unmapping init [mem 0xffff9e553cc54000-0xffff9e553ffbffff] [ 2.487373] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.490051] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.494608] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.002730] Initialise system trusted keyrings [ 3.004595] Key type blacklist registered [ 3.007910] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.020069] zbud: loaded [ 3.024287] *** VALIDATE nfs *** [ 3.025658] *** VALIDATE nfs4 *** [ 3.027272] pstore: using deflate compression [ 3.031553] Platform Keyring initialized [ 3.129441] NET: Registered protocol family 38 [ 3.134694] Key type asymmetric registered [ 3.136609] Asymmetric key parser 'x509' registered [ 3.138779] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.142127] io scheduler mq-deadline registered [ 3.143754] io scheduler kyber registered [ 3.145862] io scheduler bfq registered [ 3.148201] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.153694] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.156847] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.159878] ACPI: Power Button [PWRF] [ 3.166048] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.173540] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.186727] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.194037] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.207406] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.234863] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.266274] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.272586] Non-volatile memory driver v1.3 [ 3.274572] Linux agpgart interface v0.103 [ 3.309996] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.313266] vda: detected capacity change from 0 to 74690560 [ 3.327677] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.331203] vdb: detected capacity change from 0 to 1073741824 [ 3.347922] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.351706] vdc: detected capacity change from 0 to 2621440000 [ 3.367830] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.370773] vdd: detected capacity change from 0 to 2621440000 [ 3.387060] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.390273] vde: detected capacity change from 0 to 4294967296 [ 3.406882] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.409758] vdf: detected capacity change from 0 to 4294967296 [ 3.415918] libphy: Fixed MDIO Bus: probed [ 3.420927] usbcore: registered new interface driver usbserial_generic [ 3.423755] usbserial: USB Serial support registered for generic [ 3.426242] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.430656] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.432681] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.435227] mousedev: PS/2 mouse device common for all mice [ 3.438857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.439319] rtc_cmos 00:05: RTC can wake from S4 [ 3.446223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.447411] rtc_cmos 00:05: registered as rtc0 [ 3.451863] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.453632] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.456622] intel_pstate: CPU model not supported [ 3.462142] hid: raw HID events driver (C) Jiri Kosina [ 3.464435] usbcore: registered new interface driver usbhid [ 3.466736] usbhid: USB HID core driver [ 3.468465] drop_monitor: Initializing network drop monitor service [ 3.471202] Initializing XFRM netlink socket [ 3.473516] NET: Registered protocol family 10 [ 3.476551] Segment Routing with IPv6 [ 3.478812] NET: Registered protocol family 17 [ 3.480978] mpls_gso: MPLS GSO support [ 3.487512] RAS: Correctable Errors collector initialized. [ 3.490079] AVX version of gcm_enc/dec engaged. [ 3.491886] AES CTR mode by8 optimization enabled [ 3.565467] sched_clock: Marking stable (3565443923, 0)->(4532562569, -967118646) [ 3.568802] registered taskstats version 1 [ 3.571285] Loading compiled-in X.509 certificates [ 3.574215] zswap: loaded using pool lzo/zbud [ 3.596038] Key type big_key registered [ 3.608253] Key type encrypted registered [ 3.610264] ima: No TPM chip found, activating TPM-bypass! [ 3.612841] ima: Allocated hash algorithm: sha1 [ 3.614776] ima: No architecture policies found [ 3.616661] evm: Initialising EVM extended attributes: [ 3.618318] evm: security.selinux [ 3.619516] evm: security.ima [ 3.620794] evm: security.capability [ 3.622275] evm: HMAC attrs: 0x1 [ 3.624833] rtc_cmos 00:05: setting system clock to 2026-07-29 07:25:42 UTC (1785309942) [ 3.631403] debug: unmapping init [mem 0xffffffffa1403000-0xffffffffa15fffff] [ 3.634371] debug: unmapping init [mem 0xffffffffa0182000-0xffffffffa0458fff] [ 3.642107] Write protecting the kernel read-only data: 28672k [ 3.645334] debug: unmapping init [mem 0xffffffff9e803000-0xffffffff9e9fffff] [ 3.648251] debug: unmapping init [mem 0xffffffff9f114000-0xffffffff9f1fffff] [ 3.684606] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.693468] systemd[1]: Detected virtualization kvm. [ 3.694875] systemd[1]: Detected architecture x86-64. [ 3.696076] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.718849] systemd[1]: No hostname configured. [ 3.720814] systemd[1]: Set hostname to . [ 3.723414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.727116] systemd[1]: Initializing machine ID from random generator. [ 3.774680] random: ln: uninitialized urandom read (6 bytes read) [ 3.892039] random: systemd: uninitialized urandom read (16 bytes read) [ 3.894974] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.906290] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.910995] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.550495] device-mapper: uevent: version 1.0.3 [ 4.552716] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.352366] virtio_net virtio0 ens2: renamed from eth0 [ 5.365032] random: fast init done [ 5.425528] scsi host0: ata_piix [ 5.469642] scsi host1: ata_piix [ 5.471805] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.474351] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.879248] dracut-initqueue[598]: RTNETLINK answers: File exists [ 10.127227] random: crng init done [ 10.128741] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.666072] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.916495] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.179285] SELinux: Disabled at runtime. [ 12.239949] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.246935] systemd[1]: Detected virtualization kvm. [ 12.248555] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.794586] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.797886] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.803659] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.809321] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.812859] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.820401] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.827613] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached targ[ 12.882462] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.270054] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.574067] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.604376] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.767547] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.776513] EDAC sbridge: Ver: 1.1.2 [ 15.564104] Key type dns_resolver registered [ 15.889123] NFS: Registering the id_resolver key type [ 15.891504] Key type id_resolver registered [ 15.893483] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg638-server login: [ 41.921691] libcfs: loading out-of-tree module taints kernel. [ 41.945168] Key type ._llcrypt registered [ 41.946248] Key type .llcrypt registered [ 41.998851] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 50.924453] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 51.710450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.718591] alg: No test for adler32 (adler32-zlib) [ 52.775800] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 53.161326] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 54.799210] Key type lgssc registered [ 56.980093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.043590] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.833350] hrtimer: interrupt took 11214820 ns [ 116.081830] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 129.493949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.546525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 130.991618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.061216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.226298] Lustre: lustre-MDT0000: new disk, initializing [ 131.387384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.407154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 136.695261] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.568313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.700517] Lustre: 6496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 150.726735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 150.730416] Lustre: Skipped 1 previous similar message [ 150.792836] Lustre: lustre-MDT0001: new disk, initializing [ 150.833097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.869067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 150.884319] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 155.918929] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.461619] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 171.244651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.556839] Lustre: lustre-OST0000: new disk, initializing [ 171.560307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 171.568570] Lustre: 8434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 171.639739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.731303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.740286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 175.799119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 177.469452] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.294606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.439377] Lustre: lustre-OST0001: new disk, initializing [ 191.443936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.449978] Lustre: 9506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.529982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.995768] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.229466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.243746] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 200.282687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 209.763613] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.049468] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 223.165483] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing check_logdir /tmp/testlogs/ [ 226.969575] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing yml_node [ 230.680614] Lustre: DEBUG MARKER: Client: 2.17.55.22 [ 233.055695] Lustre: DEBUG MARKER: MDS: 2.17.55.22 [ 235.825238] Lustre: DEBUG MARKER: OSS: 2.17.55.22 [ 237.526554] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 29 03:29:34 EDT 2026 [ 251.848037] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 253.256701] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 268.768761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 268.783706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.795223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.795401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.796750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.809203] Lustre: Skipped 1 previous similar message [ 269.817229] Lustre: Skipped 1 previous similar message [ 274.412526] Lustre: server umount lustre-MDT0000 complete [ 277.584273] LustreError: 6487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310216 with bad export cookie 587900414725872152 [ 277.587101] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 277.593618] LustreError: 6487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 280.032127] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.038268] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.051816] Lustre: Skipped 1 previous similar message [ 280.065282] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 280.082401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.081268] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.120779] LustreError: 6503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 284.093444] Lustre: server umount lustre-MDT0001 complete [ 287.677981] Lustre: server umount lustre-OST0000 complete [ 291.075279] Lustre: server umount lustre-OST0001 complete [ 296.504311] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 303.638575] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 331.044169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 331.211720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 331.252679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 331.331302] Lustre: lustre-MDT0000: new disk, initializing [ 331.407238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 331.424896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.834805] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 348.096310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.336202] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 348.513394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 348.524238] Lustre: Skipped 1 previous similar message [ 348.710047] Lustre: lustre-MDT0001: new disk, initializing [ 348.875775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 348.941482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 348.953308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 355.282967] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 365.813347] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 371.360823] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 378.797581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 379.141429] Lustre: lustre-OST0000: new disk, initializing [ 379.148748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 379.162621] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 379.268988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.457440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 380.467123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 380.549794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 385.981507] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 395.057992] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 398.969434] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 399.147966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 402.723177] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 402.934514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 405.990485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.993469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 406.001929] Lustre: Skipped 2 previous similar messages [ 406.015801] Lustre: Skipped 4 previous similar messages [ 410.697154] Lustre: server umount lustre-OST0000 complete [ 415.713848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 415.718936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.730747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 420.849936] Lustre: server umount lustre-MDT0000 complete [ 421.351177] LustreError: 16620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 421.375749] LustreError: 16620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 423.964903] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310362 with bad export cookie 587900414725873678 [ 423.966315] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 423.973518] LustreError: 15042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 424.300849] Lustre: server umount lustre-MDT0001 complete [ 427.697764] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:32:45 (1785310365) [ 429.955478] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 431.401744] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:32:48 (1785310368) [ 438.159893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.641907] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.714654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.799988] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 443.874112] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 449.655209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 453.730439] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 460.139355] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.290248] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 471.199354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 477.000472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 483.362385] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.957734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.968127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.985475] Lustre: Skipped 1 previous similar message [ 489.992116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.995730] Lustre: Skipped 2 previous similar messages [ 494.054324] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.979894] Lustre: server umount lustre-OST0000 complete [ 501.216697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.389263] Lustre: server umount lustre-MDT0000 complete [ 508.716053] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310447 with bad export cookie 587900414725874560 [ 508.726228] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 508.727392] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 508.971163] Lustre: server umount lustre-MDT0001 complete [ 518.283820] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 520.739218] Key type lgssc unregistered [ 520.989504] LNet: 22964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.996131] LNetError: 22964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.017460] LNet: Removed LNI 192.168.206.138@tcp [ 521.870185] Key type .llcrypt unregistered [ 521.872055] Key type ._llcrypt unregistered [ 542.474615] Key type ._llcrypt registered [ 542.476656] Key type .llcrypt registered [ 542.544499] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 543.569698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.653600] alg: No test for adler32 (adler32-zlib) [ 544.618778] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 544.811676] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 546.463152] Key type lgssc registered [ 547.354704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.640322] loop: module loaded [ 556.713137] loop0: detected capacity change from 0 to 204800000 [ 557.000885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 557.013293] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 557.414793] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 564.001813] loop0: detected capacity change from 0 to 204800000 [ 564.360662] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 564.372182] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 564.383795] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 568.800865] loop0: detected capacity change from 0 to 204800000 [ 574.448197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 574.487806] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.816575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 575.892774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 576.019946] Lustre: lustre-MDT0000: new disk, initializing [ 576.150441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 576.182732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 580.802833] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 588.694941] loop1: detected capacity change from 0 to 204800000 [ 593.466696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 593.757793] Lustre: lustre-OST07c6: new disk, initializing [ 593.762702] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 593.767102] Lustre: Skipped 1 previous similar message [ 593.776153] Lustre: 26149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 593.886811] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 599.654943] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 600.736424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 600.757894] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 600.818345] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 605.182193] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 605.192570] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 605.202596] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 605.212780] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 605.223913] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 605.232258] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 619.531479] Lustre: 25137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 619.540395] Lustre: 25137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 619.547867] Lustre: 25137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 619.553996] Lustre: 25137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 619.558806] Lustre: 25137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 619.562911] Lustre: 25137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 624.411949] Lustre: 26157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 624.423731] Lustre: 26157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 624.440191] Lustre: 26157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 624.448300] Lustre: 26157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 624.456503] Lustre: 26157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 624.472065] Lustre: 26157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 624.481950] Lustre: 26157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 624.490924] Lustre: 26157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 624.496594] Lustre: 26157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 624.504581] Lustre: 26157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 624.510711] Lustre: 26157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 624.522573] Lustre: 26157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 627.713477] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 627.720840] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 627.726295] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 627.731516] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 627.736376] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 627.741231] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 627.748454] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 627.753870] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 627.758855] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 627.763388] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 627.768069] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 627.772281] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.413688] Lustre: 26157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 632.428749] Lustre: 26157:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 632.440587] Lustre: 26157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 632.453023] Lustre: 26157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.459307] Lustre: 26157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 632.466364] Lustre: 26157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.475573] Lustre: 26157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 632.487425] Lustre: 26157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.491584] Lustre: 26157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 632.495943] Lustre: 26157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 632.501651] Lustre: 26157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 632.509354] Lustre: 26157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 640.440282] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 640.453673] Lustre: 25136:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 640.465749] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 640.476162] Lustre: 25136:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 640.480173] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 640.486397] Lustre: 25136:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 640.492559] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 640.498738] Lustre: 25136:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 640.504526] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 640.508742] Lustre: 25136:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 640.513822] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 640.519930] Lustre: 25136:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 692.709891] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.736035] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 697.728876] Lustre: server umount lustre-OST07c6 complete [ 706.120610] Lustre: server umount lustre-MDT0000 complete [ 721.227795] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 723.792057] Key type lgssc unregistered [ 724.108599] LNet: 28708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.120092] LNetError: 28708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.136048] LNet: Removed LNI 192.168.206.138@tcp [ 724.799383] Key type .llcrypt unregistered [ 724.802289] Key type ._llcrypt unregistered [ 743.470946] Key type ._llcrypt registered [ 743.475463] Key type .llcrypt registered [ 743.562204] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 744.537986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.564350] alg: No test for adler32 (adler32-zlib) [ 745.632032] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 745.939656] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 747.647287] Key type lgssc registered [ 748.646293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.712626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.743037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.131298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.072706] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 766.442897] LustreError: 29982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.454422] LustreError: 29982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 768.958091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.198673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 773.314838] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 781.468167] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 787.278458] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 794.057123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.405847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.268231] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 801.588068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 807.627899] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 825.320109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.324275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.328494] Lustre: Skipped 1 previous similar message [ 825.346440] Lustre: Skipped 1 previous similar message [ 829.387406] Lustre: server umount lustre-OST0000 complete [ 835.553795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.556606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.569472] Lustre: Skipped 1 previous similar message [ 838.548231] Lustre: server umount lustre-MDT0000 complete [ 840.672129] LustreError: 30708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.692432] LustreError: 30708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 842.410800] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310781 with bad export cookie 1524870986942672749 [ 842.426344] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 842.427158] LustreError: 29961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 842.786659] Lustre: server umount lustre-MDT0001 complete [ 852.816712] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 855.203750] Key type lgssc unregistered [ 855.505678] LNet: 33714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.510554] LNetError: 33714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.527694] LNet: Removed LNI 192.168.206.138@tcp [ 856.392450] Key type .llcrypt unregistered [ 856.395676] Key type ._llcrypt unregistered [ 874.157607] Key type ._llcrypt registered [ 874.160163] Key type .llcrypt registered [ 874.238385] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 875.070984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.182289] alg: No test for adler32 (adler32-zlib) [ 876.242493] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 876.441200] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 878.087157] Key type lgssc registered [ 879.041858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 886.361615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 886.382479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.839975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.431351] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 898.017211] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.032815] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 899.283378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.619407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 903.779499] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 911.340139] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 916.973808] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 923.503925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.796519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.808634] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 933.056544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 938.215573] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 950.761386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 950.779564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.807685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.829891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.838568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.846055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.866143] Lustre: Skipped 1 previous similar message [ 955.591768] Lustre: server umount lustre-OST0000 complete [ 958.956611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.959949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 958.974232] Lustre: Skipped 1 previous similar message [ 958.987839] Lustre: Skipped 1 previous similar message [ 964.065044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.072851] Lustre: Skipped 1 previous similar message [ 969.184934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.198969] Lustre: Skipped 1 previous similar message [ 973.279271] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 973.426607] Lustre: server umount lustre-MDT0000 complete [ 974.308480] LustreError: 37284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.320393] LustreError: 37284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 976.526643] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785310915 with bad export cookie 13144890462631518244 [ 976.531894] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.537034] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 976.841411] Lustre: server umount lustre-MDT0001 complete [ 985.674276] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 987.914251] Key type lgssc unregistered [ 988.124570] LNet: 38524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.130800] LNetError: 38524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.157713] LNet: Removed LNI 192.168.206.138@tcp [ 988.790519] Key type .llcrypt unregistered [ 988.792927] Key type ._llcrypt unregistered [ 1009.117655] Key type ._llcrypt registered [ 1009.121560] Key type .llcrypt registered [ 1009.251845] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1010.154176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1010.232601] alg: No test for adler32 (adler32-zlib) [ 1011.195846] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1011.388115] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1013.039416] Key type lgssc registered [ 1013.888865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.599599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.624302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.090060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.283599] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1031.652824] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.675129] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1034.054590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.326260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.432945] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1045.135741] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.925560] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1055.299226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.571782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.672252] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1066.859964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1067.391575] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1074.203290] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1074.988793] Lustre: server umount lustre-MDT0000 complete [ 1075.168611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.175663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.188977] LustreError: 39940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.217931] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.218230] LustreError: 40668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.251340] LustreError: 40668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1078.153564] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311017 with bad export cookie 3515571367272658122 [ 1078.158654] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1078.163532] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1079.309897] LustreError: 42352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.340042] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 1082.345549] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.358281] Lustre: Skipped 1 previous similar message [ 1082.361529] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1083.360409] LustreError: 39935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.382824] LustreError: 39935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1084.547957] Lustre: server umount lustre-MDT0001 complete [ 1113.055153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1113.206464] Lustre: server umount lustre-OST0000 complete [ 1124.827717] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1126.866853] Key type lgssc unregistered [ 1127.131124] LNet: 43797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.140460] LNetError: 43797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.158214] LNet: Removed LNI 192.168.206.138@tcp [ 1127.937163] Key type .llcrypt unregistered [ 1127.940180] Key type ._llcrypt unregistered [ 1144.976857] Key type ._llcrypt registered [ 1144.982252] Key type .llcrypt registered [ 1145.065445] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1145.867274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1145.895896] alg: No test for adler32 (adler32-zlib) [ 1146.871653] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1147.046038] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1148.695219] Key type lgssc registered [ 1149.810987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.797543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.825835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.232717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.664801] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1167.177133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.468545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1170.980609] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1176.985653] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.456462] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1186.921421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.182446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.206198] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1196.278412] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1203.564497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1211.927200] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1212.897933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.903267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.918071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.923509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1213.924118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.924596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.957117] Lustre: Skipped 1 previous similar message [ 1216.199883] Lustre: server umount lustre-OST0000 complete [ 1233.890339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.894384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.903083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.508705] Lustre: server umount lustre-MDT0000 complete [ 1238.502251] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.516550] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1239.974945] LustreError: 45173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311178 with bad export cookie 1757119855178935068 [ 1239.980693] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.367251] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1254.555600] Lustre: server umount lustre-MDT0001 complete [ 1261.926168] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1263.954335] Key type lgssc unregistered [ 1264.211254] LNet: 48921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.225338] LNetError: 48921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.244311] LNet: Removed LNI 192.168.206.138@tcp [ 1264.879442] Key type .llcrypt unregistered [ 1264.882130] Key type ._llcrypt unregistered [ 1282.872865] Key type ._llcrypt registered [ 1282.874905] Key type .llcrypt registered [ 1282.952433] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1283.624196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1283.658717] alg: No test for adler32 (adler32-zlib) [ 1284.682889] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1284.922520] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1286.543184] Key type lgssc registered [ 1287.317735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.612115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.631883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.931936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.537683] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1302.437723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.717993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1305.476326] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1310.017972] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1313.299667] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1317.159497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1317.339541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.386648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1320.975745] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1325.393748] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.546334] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1332.676754] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1344.078551] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-29 3h48m01s [ 1345.148375] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1348.067487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.074949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.076473] Lustre: Skipped 2 previous similar messages [ 1348.087299] Lustre: Skipped 1 previous similar message [ 1348.636820] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1348.644059] Lustre: Skipped 1 previous similar message [ 1349.703599] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 1349.712408] Lustre: Skipped 3 previous similar messages [ 1352.572364] Lustre: server umount lustre-MDT0000 complete [ 1354.764574] LustreError: 52730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.779659] LustreError: 52730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1354.834057] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1359.107486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.185839] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1359.347315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.378993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1360.365745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1362.291201] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1364.785917] Lustre: DEBUG MARKER: Wait for df (36688) ... [ 1365.876556] Lustre: DEBUG MARKER: done [ 1368.546826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1368.550900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.557827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.579853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.587739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.533922] Lustre: server umount lustre-OST0000 complete [ 1380.835242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.842921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.844207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.850905] Lustre: Skipped 1 previous similar message [ 1380.854691] Lustre: Skipped 3 previous similar messages [ 1383.223569] Lustre: server umount lustre-MDT0000 complete [ 1385.546331] LustreError: 50312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311324 with bad export cookie 8263935199663264572 [ 1385.552151] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.553463] LustreError: 50312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1385.746140] Lustre: server umount lustre-MDT0001 complete [ 1391.605859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1393.313700] Key type lgssc unregistered [ 1393.506535] LNet: 55250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.517076] LNetError: 55250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.527325] LNet: Removed LNI 192.168.206.138@tcp [ 1394.059770] Key type .llcrypt unregistered [ 1394.062268] Key type ._llcrypt unregistered [ 1417.124430] Key type ._llcrypt registered [ 1417.126017] Key type .llcrypt registered [ 1417.193390] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1417.713369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.771306] alg: No test for adler32 (adler32-zlib) [ 1418.685155] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1418.867713] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1420.471129] Key type lgssc registered [ 1421.147913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1425.843815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1425.857836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1427.048369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1427.062143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1427.156488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.366402] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1434.265651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.336751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1434.358593] Lustre: 57436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1434.522780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.994983] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1441.165611] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1444.455068] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1448.537565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.660328] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1448.773842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.286200] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1455.665433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1456.550848] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1462.328917] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-29 3h50m00s [ 1463.393844] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1467.793777] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1469.288717] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1469.321323] LustreError: 57443:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1489.435309] Lustre: lustre-MDT0000: Client 2f870743-4785-46dd-8e81-fa4eb64a7d53 (at 192.168.206.38@tcp) reconnecting [ 1509.375988] LustreError: 57443:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1510.768222] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1512.251201] Lustre: DEBUG MARKER: done [ 1517.026341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.029109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.032718] LustreError: Skipped 1 previous similar message [ 1517.034357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.041922] Lustre: Skipped 1 previous similar message [ 1517.050195] Lustre: Skipped 1 previous similar message [ 1521.382264] Lustre: server umount lustre-OST0000 complete [ 1526.752842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1526.757327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.767220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.842655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.847138] Lustre: Skipped 2 previous similar messages [ 1529.605672] Lustre: server umount lustre-MDT0000 complete [ 1531.579094] LustreError: 57428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311470 with bad export cookie 12558393998500600400 [ 1531.582344] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1531.586038] LustreError: 57428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1531.751431] Lustre: server umount lustre-MDT0001 complete [ 1537.839615] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1539.432612] Key type lgssc unregistered [ 1539.603086] LNet: 61758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.614415] LNetError: 61758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.630514] LNet: Removed LNI 192.168.206.138@tcp [ 1540.114490] Key type .llcrypt unregistered [ 1540.116384] Key type ._llcrypt unregistered [ 1564.087646] Key type ._llcrypt registered [ 1564.089679] Key type .llcrypt registered [ 1564.156934] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1564.786686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1564.804834] alg: No test for adler32 (adler32-zlib) [ 1565.762039] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1565.908718] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1567.519140] Key type lgssc registered [ 1568.205613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.188053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1570.201324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.364249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1571.380435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1571.468160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1572.001331] Lustre: Failing over lustre-MDT0000 [ 1572.150470] Lustre: server umount lustre-MDT0000 complete [ 1575.305813] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:51:53 (1785311513) [ 1578.834663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.094153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.813723] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1587.000319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.064189] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1587.087870] Lustre: 64532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1587.267969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.624279] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1593.781760] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1597.135146] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1601.171362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.284473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1601.429705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.886311] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1608.282509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 1609.274902] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1613.210667] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1617.888347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1617.892042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.903237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.405060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.405929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.412461] Lustre: Skipped 1 previous similar message [ 1618.415230] Lustre: Skipped 1 previous similar message [ 1619.190541] Lustre: server umount lustre-MDT0000 complete [ 1621.499089] LustreError: 66396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311560 with bad export cookie 13485009391037276875 [ 1621.505606] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1621.507307] LustreError: 66396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1621.675588] Lustre: server umount lustre-MDT0001 complete [ 1623.620751] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1626.350485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1631.536895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.649620] LustreError: 63454:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003b9d7633 x1872034795652352/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1631.885366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.958412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1634.507170] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1637.347322] LustreError: 66379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.348726] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.363942] LustreError: 66379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1638.432958] LustreError: 66380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.437754] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1638.446574] LustreError: 66380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1639.391465] Lustre: 63456:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311562/real 1785311562] req@0000000087c1e60b x1872034795651968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1639.537704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.133580] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1642.849876] Lustre: 63457:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311565/real 1785311565] req@00000000c9445ad6 x1872034795652096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311581 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1644.895107] Lustre: 63458:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311567/real 1785311567] req@0000000058301f61 x1872034795652224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1645.033926] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1646.247110] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1648.098816] Lustre: 63455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311570/real 1785311570] req@0000000044da9a59 x1872034795652480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1649.583157] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1655.267051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.268881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1655.269668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.269672] Lustre: Skipped 1 previous similar message [ 1655.275721] Lustre: Skipped 1 previous similar message [ 1659.524964] Lustre: server umount lustre-MDT0000 complete [ 1660.388790] LustreError: 68293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.401785] LustreError: 68293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1661.705047] LustreError: 66396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311600 with bad export cookie 13485009391037279136 [ 1661.706318] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.711933] LustreError: 66396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1661.901794] Lustre: server umount lustre-MDT0001 complete [ 1665.468219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1670.507912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.647120] LustreError: 70960:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1671.647848] LustreError: 63454:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a91d6569 x1872034795675008/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1671.678226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbb24609831957103 [ 1671.698356] Lustre: MGC192.168.206.138@tcp: Connection restored to 0@lo (at 0@lo) [ 1671.879262] LustreError: 66379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1671.927835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.932141] Lustre: Skipped 1 previous similar message [ 1671.965766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1674.333498] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1677.284394] LustreError: 66380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.284487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.298384] Lustre: Skipped 1 previous similar message [ 1679.402900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.632250] Lustre: 63456:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785311604/real 1785311604] req@0000000060ac8727 x1872034795674624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785311620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1682.468425] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1684.974158] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1686.481890] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1689.401727] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1695.201073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1695.204721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.211935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.214865] Lustre: Skipped 2 previous similar messages [ 1698.739310] Lustre: server umount lustre-MDT0000 complete [ 1700.676646] LustreError: 64525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311639 with bad export cookie 13485009391037280515 [ 1700.679445] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1700.683437] LustreError: 64525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1700.869618] Lustre: server umount lustre-MDT0001 complete [ 1704.218612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1706.050314] Lustre: DEBUG MARKER: files compared the same [ 1716.294698] Lustre: server umount lustre-OST0000 complete [ 1725.254467] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1726.808735] Key type lgssc unregistered [ 1726.968612] LNet: 74247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.975503] LNetError: 74247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.986648] LNet: Removed LNI 192.168.206.138@tcp [ 1727.454672] Key type .llcrypt unregistered [ 1727.456348] Key type ._llcrypt unregistered [ 1741.525157] Key type ._llcrypt registered [ 1741.526743] Key type .llcrypt registered [ 1741.586798] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1742.048565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.114777] alg: No test for adler32 (adler32-zlib) [ 1743.081064] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1743.238106] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1744.855301] Key type lgssc registered [ 1745.580188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.425540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.442493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.722756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.334455] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 1759.387661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.568205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1761.991320] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 1766.359080] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1769.500157] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1773.048587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.201573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.209502] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug +malloc all [ 1776.297656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1780.399448] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1789.919871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.920751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.926226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.933724] Lustre: Skipped 1 previous similar message [ 1789.942942] Lustre: Skipped 1 previous similar message [ 1795.049505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.055577] Lustre: Skipped 1 previous similar message [ 1800.161796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.165376] Lustre: Skipped 1 previous similar message [ 1804.255116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1804.329287] Lustre: server umount lustre-OST0000 complete [ 1810.400408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.407495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.410591] Lustre: Skipped 1 previous similar message [ 1812.490502] Lustre: server umount lustre-MDT0000 complete [ 1814.350569] LustreError: 75623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311753 with bad export cookie 13942577548779765002 [ 1814.351736] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.361911] LustreError: 75623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1814.502650] Lustre: server umount lustre-MDT0001 complete [ 1819.695280] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1821.193877] Key type lgssc unregistered [ 1821.369397] LNet: 79174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.374536] LNetError: 79174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.385387] LNet: Removed LNI 192.168.206.138@tcp [ 1821.842476] Key type .llcrypt unregistered [ 1821.846330] Key type ._llcrypt unregistered [ 1833.880043] Key type ._llcrypt registered [ 1833.881814] Key type .llcrypt registered [ 1833.933672] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1834.476557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.487765] alg: No test for adler32 (adler32-zlib) [ 1835.419932] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1835.566135] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1837.191172] Key type lgssc registered [ 1837.798669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.713079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.726835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1857.247319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1862.367294] LustreError: 80542:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 1889.055456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.102153] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1895.683075] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1899.304898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.370568] Lustre: *** cfs_fail_loc=706, val=0*** [ 1899.586916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.628406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1901.543829] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1905.566150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.762465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1908.001408] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1911.670335] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1914.249921] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.872988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1915.877753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.886479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.389756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1916.390399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.508800] Lustre: server umount lustre-OST0000 complete [ 1926.112230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.116148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.125706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.130139] Lustre: Skipped 1 previous similar message [ 1929.330786] Lustre: server umount lustre-MDT0000 complete [ 1931.100326] LustreError: 80550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785311869 with bad export cookie 5194914989058487927 [ 1931.103880] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.108175] LustreError: 80550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1931.238873] Lustre: server umount lustre-MDT0001 complete [ 1936.031286] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 1937.420737] Key type lgssc unregistered [ 1937.594475] LNet: 84152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.597948] LNetError: 84152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.608612] LNet: Removed LNI 192.168.206.138@tcp [ 1938.049726] Key type .llcrypt unregistered [ 1938.051514] Key type ._llcrypt unregistered [ 1950.921690] Key type ._llcrypt registered [ 1950.923831] Key type .llcrypt registered [ 1950.980787] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 1951.452799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.487857] alg: No test for adler32 (adler32-zlib) [ 1952.393341] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 1952.513669] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 1954.127145] Key type lgssc registered [ 1954.649813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1958.147530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1958.156512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.157459] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1964.649255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.906157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.749085] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1970.144338] LustreError: 86232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.154672] LustreError: 86232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1970.391379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1970.551157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.025176] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1974.625490] LustreError: 86236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.635241] LustreError: 86236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1976.918483] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.925585] Lustre: MGS: Client 0bcec396-58ad-44ad-acfb-468e32c75792 (at 0@lo) reconnecting [ 1976.937741] LustreError: 86246:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1977.051098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.085874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1978.834322] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 1987.554074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.554347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.560762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.566835] Lustre: Skipped 1 previous similar message [ 1992.675576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1992.678073] Lustre: Skipped 1 previous similar message [ 1993.491223] Lustre: server umount lustre-OST0000 complete [ 1997.794152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.795055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.799474] Lustre: Skipped 2 previous similar messages [ 1997.805741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2000.971875] Lustre: server umount lustre-MDT0000 complete [ 2002.617560] Lustre: server umount lustre-MDT0001 complete [ 2004.259869] Lustre: server umount MGS complete [ 2009.025309] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:59:07 (1785311947) [ 2021.130852] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2025.039393] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2039.945450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.608456] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2046.697844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.805076] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2046.835643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2046.877336] Lustre: lustre-MDT0001: new disk, initializing [ 2046.916850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2046.927994] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2046.931691] Lustre: 93386:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2046.937200] LustreError: 93386:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2048.657171] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2053.689747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.811873] Lustre: lustre-OST0000: new disk, initializing [ 2053.814290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2053.817722] Lustre: 94153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2053.848297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.195129] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2057.184074] Lustre: 94486:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c16b735e x1872035201451264/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785312012 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2060.705658] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2063.397787] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.403435] Lustre: MGS: Client 56c64e6d-04ef-432c-8f4b-4b20c75d6970 (at 0@lo) reconnecting [ 2063.421707] Lustre: 95199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2063.433430] LustreError: 93372:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2063.438133] LustreError: 93372:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2063.443614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2063.484495] Lustre: lustre-MDT0000: new disk, initializing [ 2063.544682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.557620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2064.607485] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2064.611530] Lustre: 95224:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009de2dd88 x1872035201457280/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785312019 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2065.169404] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2066.281319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2066.286414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2066.733634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2084.320360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2084.324492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.329873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2084.333078] Lustre: Skipped 1 previous similar message [ 2090.070329] Lustre: server umount lustre-OST0000 complete [ 2096.096679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.096740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.097701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.097705] Lustre: Skipped 2 previous similar messages [ 2096.109108] Lustre: Skipped 2 previous similar messages [ 2097.668713] Lustre: server umount lustre-MDT0000 complete [ 2105.064765] Lustre: server umount lustre-MDT0001 complete [ 2106.375791] Lustre: server umount MGS complete [ 2110.349236] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:00:48 (1785312048) [ 2116.747568] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2117.869801] Key type lgssc unregistered [ 2118.008400] LNet: 97715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.011421] LNetError: 97715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.020367] LNet: Removed LNI 192.168.206.138@tcp [ 2118.352658] Key type .llcrypt unregistered [ 2118.354479] Key type ._llcrypt unregistered [ 2126.197611] Key type ._llcrypt registered [ 2126.199346] Key type .llcrypt registered [ 2126.251379] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2126.656573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.690162] alg: No test for adler32 (adler32-zlib) [ 2127.569491] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2127.677320] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2129.263221] Key type lgssc registered [ 2129.715138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.498295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.504212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.647072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.015969] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2137.936743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.068310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.527231] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2142.402299] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2144.594179] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2147.267086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2147.390161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.560180] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2152.209777] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2158.693757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2168.290517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.292608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.294621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.298724] Lustre: Skipped 1 previous similar message [ 2168.302760] Lustre: Skipped 1 previous similar message [ 2172.636820] Lustre: server umount lustre-OST0000 complete [ 2178.528418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.529139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.533279] Lustre: Skipped 1 previous similar message [ 2183.648148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.650365] Lustre: Skipped 2 previous similar messages [ 2188.255189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2188.323302] Lustre: server umount lustre-MDT0000 complete [ 2188.768495] LustreError: 101709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.777561] LustreError: 101709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2189.728928] LustreError: 98911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312128 with bad export cookie 9109231649827633035 [ 2189.730487] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.734110] LustreError: 98911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2189.864636] Lustre: server umount lustre-MDT0001 complete [ 2194.010304] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2195.210225] Key type lgssc unregistered [ 2195.349862] LNet: 102482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.354426] LNetError: 102482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.364418] LNet: Removed LNI 192.168.206.138@tcp [ 2195.714920] Key type .llcrypt unregistered [ 2195.716442] Key type ._llcrypt unregistered [ 2202.526392] Key type ._llcrypt registered [ 2202.527623] Key type .llcrypt registered [ 2202.570083] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2202.975720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.017068] alg: No test for adler32 (adler32-zlib) [ 2203.889297] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2204.003386] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2205.599256] Key type lgssc registered [ 2206.019995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2208.294104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2208.294366] LustreError: 103543:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2208.298663] Lustre: Skipped 1 previous similar message [ 2208.302927] LustreError: 103542:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2208.307569] LustreError: 103543:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2208.310579] LustreError: 103543:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2208.317429] LustreError: 103543:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2208.320608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.501382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.329848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.426710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2213.897691] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2215.778830] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lsmod [ 2216.859101] LustreError: 104747:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2216.859121] LustreError: 104748:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2216.865194] LustreError: 104747:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 2216.868160] LustreError: 104747:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2216.871459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.873309] LustreError: 104747:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2217.015641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.050511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.052100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.055632] Lustre: Skipped 2 previous similar messages [ 2222.058300] Lustre: Skipped 1 previous similar message [ 2223.073698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.076096] Lustre: Skipped 3 previous similar messages [ 2228.193596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.196329] Lustre: Skipped 2 previous similar messages [ 2232.287176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2232.347880] Lustre: server umount lustre-MDT0000 complete [ 2233.312831] LustreError: 103580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.319645] LustreError: 103580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2233.725431] LustreError: 103558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312172 with bad export cookie 17063856333017789504 [ 2233.728434] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.730508] LustreError: 103558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2235.359255] LustreError: 104011:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006f484769 x1872035464628352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2235.408826] Lustre: server umount lustre-MDT0001 complete [ 2246.688252] Lustre: server umount lustre-OST0000 complete [ 2249.747522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2249.934561] LustreError: 105812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.967472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.480889] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2254.607445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.753967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2256.193202] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2258.812141] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2260.804199] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.150298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.265560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.250114] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2267.816976] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2274.539075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2279.907192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.908484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.909240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.912454] Lustre: Skipped 1 previous similar message [ 2279.916462] LustreError: Skipped 1 previous similar message [ 2285.025244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.027672] Lustre: Skipped 2 previous similar messages [ 2294.239162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2294.296073] Lustre: server umount lustre-OST0000 complete [ 2295.775660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2295.778154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.783444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.785940] Lustre: Skipped 2 previous similar messages [ 2300.384786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.776680] Lustre: server umount lustre-MDT0000 complete [ 2303.088390] LustreError: 105794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312241 with bad export cookie 17063856333017790190 [ 2303.090645] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.093464] LustreError: 105794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2303.218569] Lustre: server umount lustre-MDT0001 complete [ 2306.993977] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2308.060596] Key type lgssc unregistered [ 2308.185386] LNet: 109369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.189200] LNetError: 109369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.197328] LNet: Removed LNI 192.168.206.138@tcp [ 2308.514947] Key type .llcrypt unregistered [ 2308.516852] Key type ._llcrypt unregistered [ 2318.146438] Key type ._llcrypt registered [ 2318.148189] Key type .llcrypt registered [ 2318.193756] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2318.513159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.520493] alg: No test for adler32 (adler32-zlib) [ 2319.393662] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2319.499400] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2321.087329] Key type lgssc registered [ 2321.519253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2324.617842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.626550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.797669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.353508] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2330.497836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2330.635660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2332.028789] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2334.627254] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2336.686751] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.237397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.361535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.530605] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2344.243172] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2351.654886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2357.738975] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2358.562057] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2360.799969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.802765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.806683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.826461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.826715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2361.827987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.715713] Lustre: server umount lustre-OST0000 complete [ 2371.552160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.555102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2371.559586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.562633] Lustre: Skipped 1 previous similar message [ 2377.184496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.187576] Lustre: Skipped 3 previous similar messages [ 2381.279247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2381.340375] Lustre: server umount lustre-MDT0000 complete [ 2382.303902] LustreError: 110761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.309131] LustreError: 110761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2382.613903] LustreError: 110742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312321 with bad export cookie 15538278724436369790 [ 2382.617916] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.618126] LustreError: 110742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2382.738082] Lustre: server umount lustre-MDT0001 complete [ 2386.570446] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2387.598165] Key type lgssc unregistered [ 2387.727449] LNet: 114415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.731188] LNetError: 114415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.745336] LNet: Removed LNI 192.168.206.138@tcp [ 2388.094994] Key type .llcrypt unregistered [ 2388.096565] Key type ._llcrypt unregistered [ 2395.491465] Key type ._llcrypt registered [ 2395.493209] Key type .llcrypt registered [ 2395.536348] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2395.872336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.897269] alg: No test for adler32 (adler32-zlib) [ 2396.763865] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2396.863528] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2398.447169] Key type lgssc registered [ 2398.879884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2401.765506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2401.773686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.934244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.352426] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2407.447794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.585193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2409.130471] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2411.790066] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2413.868899] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2416.504643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.632620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.777134] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2421.370404] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.879259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2438.113839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2438.114634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.117170] LustreError: Skipped 1 previous similar message [ 2438.117765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2438.123853] Lustre: Skipped 1 previous similar message [ 2440.726338] Lustre: server umount lustre-OST0000 complete [ 2443.232817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2443.233354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.238965] Lustre: Skipped 1 previous similar message [ 2443.241618] Lustre: Skipped 1 previous similar message [ 2448.348708] Lustre: server umount lustre-MDT0000 complete [ 2448.353442] LustreError: 118116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.360247] LustreError: 118116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2449.647087] LustreError: 115594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312388 with bad export cookie 3170543594805192777 [ 2449.649750] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.651539] LustreError: 115594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2449.763779] Lustre: server umount lustre-MDT0001 complete [ 2453.318857] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2454.374770] Key type lgssc unregistered [ 2454.510457] LNet: 119169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.513708] LNetError: 119169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.526347] LNet: Removed LNI 192.168.206.138@tcp [ 2454.850208] Key type .llcrypt unregistered [ 2454.852039] Key type ._llcrypt unregistered [ 2464.476176] Key type ._llcrypt registered [ 2464.477619] Key type .llcrypt registered [ 2464.519178] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2464.945475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.963022] alg: No test for adler32 (adler32-zlib) [ 2465.828168] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2465.929786] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2467.519094] Key type lgssc registered [ 2467.921333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2470.613923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2470.622378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.800675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.251367] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2476.411114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.560782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.141853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2480.863528] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2483.025262] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2485.558316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2485.713118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2488.020490] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2490.833444] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2498.026346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 2503.692702] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2505.014808] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2505.018209] Lustre: Skipped 2 previous similar messages [ 2506.581234] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2506.584827] Lustre: Skipped 2 previous similar messages [ 2511.865568] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2511.876105] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2517.408141] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2517.412798] Lustre: Skipped 2 previous similar messages [ 2518.588642] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.38@tcp 192.168.0.[2,10]@tcp [ 2522.353137] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.38/24@tcp in log lustre-MDT0000 [ 2522.357308] Lustre: Skipped 2 previous similar messages [ 2525.945154] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 2525.947032] Lustre: Skipped 1 previous similar message [ 2528.737344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.738429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.739058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.741925] Lustre: Skipped 1 previous similar message [ 2533.857427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2533.861798] Lustre: Skipped 2 previous similar messages [ 2534.029422] Lustre: server umount lustre-OST0000 complete [ 2537.951978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.955071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.960688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.977623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.464593] Lustre: server umount lustre-MDT0000 complete [ 2542.658983] LustreError: 120543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312481 with bad export cookie 1463208768686742098 [ 2542.662909] LustreError: 120543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2542.664302] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.774949] Lustre: server umount lustre-MDT0001 complete [ 2546.450584] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2547.495547] Key type lgssc unregistered [ 2547.623389] LNet: 125506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.625978] LNetError: 125506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2547.635170] LNet: Removed LNI 192.168.206.138@tcp [ 2547.951395] Key type .llcrypt unregistered [ 2547.952735] Key type ._llcrypt unregistered [ 2558.390348] Key type ._llcrypt registered [ 2558.391577] Key type .llcrypt registered [ 2558.426511] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2558.824333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.834566] alg: No test for adler32 (adler32-zlib) [ 2559.684923] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2559.768748] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2561.343219] Key type lgssc registered [ 2561.696075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.232636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.239825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.372745] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2565.376804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.383364] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.38/24@tcp [ 2566.721097] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2569.724525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.840123] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2569.850295] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.38/24@tcp [ 2569.867085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.278868] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2573.756797] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2575.664264] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2578.020324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.144170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.126318] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2582.556993] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2591.463282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2596.833508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.833818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.839694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.841529] Lustre: Skipped 1 previous similar message [ 2598.354787] Lustre: server umount lustre-OST0000 complete [ 2600.927869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.931406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.935185] Lustre: Skipped 1 previous similar message [ 2600.936757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.679834] Lustre: server umount lustre-MDT0000 complete [ 2601.952523] LustreError: 127788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.952592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.958068] LustreError: 127788:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2602.917371] LustreError: 127038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312541 with bad export cookie 15305756240904627057 [ 2602.920950] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.921441] LustreError: 127038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2603.024733] Lustre: server umount lustre-MDT0001 complete [ 2606.741512] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2607.810351] Key type lgssc unregistered [ 2607.938433] LNet: 130659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.942375] LNetError: 130659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.951390] LNet: Removed LNI 192.168.206.138@tcp [ 2608.289066] Key type .llcrypt unregistered [ 2608.290281] Key type ._llcrypt unregistered [ 2617.143365] Key type ._llcrypt registered [ 2617.144298] Key type .llcrypt registered [ 2617.185577] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2623.328921] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2623.646915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2623.676119] alg: No test for adler32 (adler32-zlib) [ 2624.531303] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2624.615955] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2626.199234] Key type lgssc registered [ 2626.620465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2638.300753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2638.306925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.392333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2639.404506] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2639.447374] Lustre: lustre-MDT0000: new disk, initializing [ 2639.470251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.476661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2640.888164] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2645.051031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.091255] Lustre: 134029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2645.115116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2645.116940] Lustre: Skipped 1 previous similar message [ 2645.149129] Lustre: lustre-MDT0001: new disk, initializing [ 2645.179055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2645.190918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2645.193746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2646.602093] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2650.242533] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2652.033738] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2654.251749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.355933] Lustre: lustre-OST0000: new disk, initializing [ 2654.357885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2654.360540] Lustre: 135998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2654.393750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.964955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2654.969309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2655.000633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2656.338324] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2660.019614] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2661.404727] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2661.478764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2662.752668] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2662.834019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2665.441356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.443402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.444269] LustreError: Skipped 1 previous similar message [ 2665.449723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.451672] Lustre: Skipped 1 previous similar message [ 2669.579731] Lustre: server umount lustre-OST0000 complete [ 2671.071814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2671.074274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.079624] Lustre: Skipped 1 previous similar message [ 2671.081922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.680109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.680646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.899555] Lustre: server umount lustre-MDT0000 complete [ 2678.155292] LustreError: 135997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312617 with bad export cookie 5059467540190040113 [ 2678.158704] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.158977] LustreError: 135997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2678.270128] Lustre: server umount lustre-MDT0001 complete [ 2681.032760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.187963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.487488] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2685.247177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.371248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.803758] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2689.435797] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2691.335055] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2693.752642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2693.897564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.994977] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2698.577258] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2700.056688] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2707.284299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2708.777203] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2708.859858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2712.390612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.460764] Lustre: lustre-OST0001: new disk, initializing [ 2712.463232] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2712.466079] Lustre: 141625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2712.499791] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2714.535102] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2717.970174] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2722.300959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2722.305079] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2722.330803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2723.139671] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2724.473707] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2724.544484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2726.281077] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e6c98255000.ost_server_uuid 50 [ 2726.750321] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e6c98255000.ost_server_uuid in FULL state after 0 sec [ 2732.515898] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.516631] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.521542] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2732.523848] LustreError: Skipped 1 previous similar message [ 2732.527322] Lustre: Skipped 2 previous similar messages [ 2734.590767] Lustre: server umount lustre-OST0001 complete [ 2737.119872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.120082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.122729] LustreError: Skipped 1 previous similar message [ 2737.127194] Lustre: Skipped 1 previous similar message [ 2737.127658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.131379] Lustre: Skipped 1 previous similar message [ 2737.634891] LustreError: 140385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.640587] LustreError: 140385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2741.893432] Lustre: server umount lustre-OST0000 complete [ 2746.847850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.850474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.853976] Lustre: Skipped 1 previous similar message [ 2746.855746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.211657] Lustre: server umount lustre-MDT0000 complete [ 2750.398576] LustreError: 138190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312689 with bad export cookie 5059467540190040981 [ 2750.402411] LustreError: 138190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2750.402758] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.506268] Lustre: server umount lustre-MDT0001 complete [ 2756.306180] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2757.252825] Key type lgssc unregistered [ 2757.379478] LNet: 144658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.381698] LNetError: 144658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.391310] LNet: Removed LNI 192.168.206.138@tcp [ 2757.653193] Key type .llcrypt unregistered [ 2757.654277] Key type ._llcrypt unregistered [ 2770.464276] Key type ._llcrypt registered [ 2770.465559] Key type .llcrypt registered [ 2770.502129] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2776.381977] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2776.789435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2776.802315] alg: No test for adler32 (adler32-zlib) [ 2777.644092] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2777.724736] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2779.303149] Key type lgssc registered [ 2779.646388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.457650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.464029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.547752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2791.559208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2791.597620] Lustre: lustre-MDT0000: new disk, initializing [ 2791.624067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2791.632350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2793.006997] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2796.971322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.013246] Lustre: 148783:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2797.036408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2797.038301] Lustre: Skipped 1 previous similar message [ 2797.076954] Lustre: lustre-MDT0001: new disk, initializing [ 2797.115697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2797.129274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2797.134057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2798.446947] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2802.102769] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.880707] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2806.017942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.116380] Lustre: lustre-OST0000: new disk, initializing [ 2806.118208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2806.121153] Lustre: 150748:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2806.148845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.042591] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2809.141422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2809.144363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2809.171771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2811.689544] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2813.620062] Lustre: Failing over lustre-OST0000 [ 2813.675152] Lustre: server umount lustre-OST0000 complete [ 2826.309279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2826.398824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.405482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2826.411800] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2827.401171] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2828.143917] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2829.329144] Lustre: Failing over lustre-MDT0000 [ 2829.418574] Lustre: server umount lustre-MDT0000 complete [ 2830.367172] Lustre: 147009:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785312753/real 1785312753] req@000000006db3fcc0 x1872036066517376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785312769 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2831.841572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.847591] Lustre: Skipped 3 previous similar messages [ 2835.487110] Lustre: 147012:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785312758/real 1785312758] req@00000000aceafad1 x1872036066519296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785312774 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2836.960189] LustreError: 148793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.968618] LustreError: 148793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2837.506082] LustreError: 151470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.055081] Lustre: 147010:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785312763/real 1785312763] req@000000009a251e45 x1872036066520448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785312779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2842.080966] LustreError: 148792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.086851] LustreError: 148792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2842.382594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.420485] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.424692] Lustre: 148785:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.138@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2842.501600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2842.522221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.545678] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2842.626220] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2843.764458] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2847.721866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2847.729377] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2847.755148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2852.834050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2852.834759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.836281] LustreError: Skipped 1 previous similar message [ 2852.840648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2852.842675] Lustre: Skipped 1 previous similar message [ 2854.798296] Lustre: server umount lustre-OST0000 complete [ 2857.953771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2857.954292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.961686] Lustre: Skipped 1 previous similar message [ 2857.964664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.966685] Lustre: Skipped 1 previous similar message [ 2862.163850] Lustre: server umount lustre-MDT0000 complete [ 2863.071936] LustreError: 148793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.076392] LustreError: 148793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2863.346411] LustreError: 148775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312802 with bad export cookie 5379892920314578751 [ 2863.349356] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2863.350548] LustreError: 148775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2863.472879] Lustre: server umount lustre-MDT0001 complete [ 2867.006540] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 2868.027812] Key type lgssc unregistered [ 2868.162333] LNet: 154398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.164475] LNetError: 154398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.173255] LNet: Removed LNI 192.168.206.138@tcp [ 2868.484426] Key type .llcrypt unregistered [ 2868.485919] Key type ._llcrypt unregistered [ 2888.280374] Key type ._llcrypt registered [ 2888.281310] Key type .llcrypt registered [ 2888.321335] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2888.813091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2888.849283] alg: No test for adler32 (adler32-zlib) [ 2889.686649] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 2889.766505] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 2891.343122] Key type lgssc registered [ 2891.716584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2894.450616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2894.458536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.624717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.085498] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2900.789798] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.806862] LustreError: 155944:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2902.811387] LustreError: 155944:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2902.814890] LustreError: 155944:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2902.925939] Lustre: server umount lustre-MDT0000 complete [ 2907.552309] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 2910.683285] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 2921.918723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.003562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2922.017593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2922.056348] Lustre: lustre-MDT0000: new disk, initializing [ 2922.079529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2922.086113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2923.483040] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2927.500772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.538709] Lustre: 159241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.560495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2927.563428] Lustre: Skipped 1 previous similar message [ 2927.600814] Lustre: lustre-MDT0001: new disk, initializing [ 2927.639064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2927.654606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2927.660514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2928.960664] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2932.469934] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2934.102719] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2936.158659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2936.248420] Lustre: lustre-OST0000: new disk, initializing [ 2936.250342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2936.252767] Lustre: 161209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2936.278054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.993301] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 2938.998777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2939.001965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2939.029048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2941.701111] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2943.571196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2944.104146] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:14:42 (1785312882) [ 3009.503576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3009.505944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.509497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.039646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.039723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3011.046125] Lustre: Skipped 1 previous similar message [ 3011.048062] Lustre: Skipped 1 previous similar message [ 3015.523544] Lustre: server umount lustre-MDT0000 complete [ 3016.159545] LustreError: 162971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3016.166535] LustreError: 162971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3016.787587] LustreError: 159232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785312955 with bad export cookie 8624411460774166916 [ 3016.789069] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3016.792966] LustreError: 159232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3021.279421] LustreError: 160532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.280510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.288051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3021.289773] Lustre: Skipped 1 previous similar message [ 3022.941354] Lustre: server umount lustre-MDT0001 complete [ 3030.346185] Lustre: server umount lustre-OST0000 complete [ 3033.403917] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3036.627853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3048.731850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.820320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3048.833388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3048.876088] Lustre: lustre-MDT0000: new disk, initializing [ 3048.898544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3048.904994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3050.418087] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3054.712283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.758468] Lustre: 165833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3054.785707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3054.787729] Lustre: Skipped 1 previous similar message [ 3054.825257] Lustre: lustre-MDT0001: new disk, initializing [ 3054.874654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3054.878554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3056.303864] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3060.042027] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3061.938973] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3064.219522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3064.338955] Lustre: lustre-OST0000: new disk, initializing [ 3064.340946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3064.343726] Lustre: 167802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3064.379976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.382208] Lustre: Skipped 1 previous similar message [ 3066.426472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3069.920788] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3070.969190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3070.972085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3071.012714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3071.250634] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3071.326302] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3072.569744] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3072.632953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3076.065627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3076.065842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.067599] LustreError: Skipped 1 previous similar message [ 3076.067960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3076.070955] Lustre: Skipped 1 previous similar message [ 3076.075026] Lustre: Skipped 1 previous similar message [ 3079.307984] Lustre: server umount lustre-OST0000 complete [ 3080.671891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3080.674162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.677522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.620796] Lustre: server umount lustre-MDT0000 complete [ 3087.880501] LustreError: 165870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313026 with bad export cookie 8624411460774325732 [ 3087.884467] LustreError: 165870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3087.886109] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.014628] Lustre: server umount lustre-MDT0001 complete [ 3090.928110] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:17:09 (1785313029) [ 3095.764603] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3098.581416] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3110.200926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.286454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3110.297929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3110.340549] Lustre: lustre-MDT0000: new disk, initializing [ 3110.364402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.370567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3111.792646] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3115.952582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.993270] Lustre: 172551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3116.023923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3116.025845] Lustre: Skipped 4 previous similar messages [ 3116.097527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3117.323279] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3120.807322] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3122.512442] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3124.526171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.628440] Lustre: 174528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3126.427605] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3128.823026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3128.856503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3129.969771] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3133.921306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3133.923532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.925200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.927290] Lustre: Skipped 2 previous similar messages [ 3137.741966] Lustre: server umount lustre-OST0000 complete [ 3153.375129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3153.439427] Lustre: server umount lustre-MDT0000 complete [ 3154.669127] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313093 with bad export cookie 8624411460774326600 [ 3154.672256] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3154.675055] LustreError: 172544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3154.795132] Lustre: server umount lustre-MDT0001 complete [ 3157.467977] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:18:15 (1785313095) [ 3162.123051] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3164.865129] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3176.222527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.313657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3176.323929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3176.326293] Lustre: Skipped 3 previous similar messages [ 3176.362814] Lustre: lustre-MDT0000: new disk, initializing [ 3176.364613] Lustre: Skipped 2 previous similar messages [ 3176.387673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.390359] Lustre: Skipped 2 previous similar messages [ 3176.396086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3176.399228] Lustre: Skipped 2 previous similar messages [ 3177.752441] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3181.912785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.957943] Lustre: 178836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3182.057353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.415949] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3187.195395] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3189.129165] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3191.525921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.646264] Lustre: 180814:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3192.351945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3193.581921] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3197.231117] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3202.530197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3202.533059] LustreError: Skipped 1 previous similar message [ 3202.534575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.538130] Lustre: Skipped 2 previous similar messages [ 3202.540174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3202.542460] Lustre: Skipped 10 previous similar messages [ 3205.134620] Lustre: server umount lustre-OST0000 complete [ 3212.443695] Lustre: server umount lustre-MDT0000 complete [ 3212.768099] LustreError: 178849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.773856] LustreError: 178849:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3213.735288] LustreError: 178830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313152 with bad export cookie 8624411460774327685 [ 3213.739360] LustreError: 178830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3213.740088] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.847222] Lustre: server umount lustre-MDT0001 complete [ 3217.636780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3218.658987] Key type lgssc unregistered [ 3218.781381] LNet: 182669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3218.785849] LNetError: 182669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3218.796333] LNet: Removed LNI 192.168.206.138@tcp [ 3219.098038] Key type .llcrypt unregistered [ 3219.099500] Key type ._llcrypt unregistered [ 3228.324606] Key type ._llcrypt registered [ 3228.326197] Key type .llcrypt registered [ 3228.364143] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3228.751425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3228.773423] alg: No test for adler32 (adler32-zlib) [ 3229.616784] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 3229.708261] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3231.287121] Key type lgssc registered [ 3231.636537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3234.216978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3234.224035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.389142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.633343] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3239.292408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.422524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.707263] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3242.981612] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.792740] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.908411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.012147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.782460] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3251.015513] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3260.326201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3275.745335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3275.747153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.748959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.758831] Lustre: Skipped 1 previous similar message [ 3278.923659] Lustre: server umount lustre-OST0000 complete [ 3280.351774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.353921] LustreError: Skipped 1 previous similar message [ 3280.355046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.358739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.360311] Lustre: Skipped 1 previous similar message [ 3285.984669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.987730] Lustre: Skipped 2 previous similar messages [ 3286.373092] Lustre: server umount lustre-MDT0000 complete [ 3287.631283] LustreError: 184797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313226 with bad export cookie 12400800654577364597 [ 3287.634745] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.634883] LustreError: 184797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3287.759108] Lustre: server umount lustre-MDT0001 complete [ 3291.610243] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3292.650967] Key type lgssc unregistered [ 3292.777366] LNet: 187618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3292.781220] LNetError: 187618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3292.787271] LNet: Removed LNI 192.168.206.138@tcp [ 3293.107277] Key type .llcrypt unregistered [ 3293.109258] Key type ._llcrypt unregistered [ 3302.521897] Key type ._llcrypt registered [ 3302.523133] Key type .llcrypt registered [ 3302.566272] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3302.931297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3302.939544] alg: No test for adler32 (adler32-zlib) [ 3303.791446] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 3303.882996] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3305.463146] Key type lgssc registered [ 3305.869942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3308.379358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3308.387628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3309.543529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.826616] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3313.538188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.656055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3314.887499] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3317.177454] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3319.043370] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3321.335228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3321.484352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.575723] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3325.990520] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.818422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3339.232353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3339.234612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.238599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.258108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3340.258125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.264913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.834365] Lustre: lustre-OST0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 3341.837012] Lustre: Skipped 1 previous similar message [ 3344.973236] Lustre: server umount lustre-OST0000 complete [ 3347.234365] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e6c82e38800.ost_server_uuid 50 [ 3347.783090] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3357.723663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.724407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.727143] Lustre: Skipped 1 previous similar message [ 3362.784099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.787032] Lustre: Skipped 2 previous similar messages [ 3363.862081] Lustre: server umount lustre-MDT0000 complete [ 3364.964218] LustreError: 188990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313303 with bad export cookie 12953599281784189225 [ 3364.967690] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.968097] LustreError: 188990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3365.074096] Lustre: server umount lustre-MDT0001 complete [ 3367.851564] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:21:46 (1785313306) [ 3370.076316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.224068] LustreError: 192922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3370.229586] LustreError: 192922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3370.245857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.539523] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3374.252522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.359910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3375.646842] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3377.899876] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3379.621116] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3381.844623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3383.739932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3386.037668] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.107684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.181882] Lustre: lustre-OST0001: new disk, initializing [ 3388.183518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3388.185849] Lustre: 195761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3388.220532] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3388.223188] Lustre: Skipped 1 previous similar message [ 3390.184941] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3393.023240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3393.026549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3393.064804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3393.970353] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3395.552050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.554492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.557847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3400.892421] Lustre: server umount lustre-OST0000 complete [ 3401.698068] LustreError: 194743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.705140] LustreError: 194743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3402.954280] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3403.029503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3404.291322] LustreError: 194746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.362423] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3404.434767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3406.819680] LustreError: 194745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3406.824607] LustreError: 194745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3411.938560] LustreError: 194745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.943287] LustreError: 194745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3417.058041] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.059859] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3417.062863] Lustre: Skipped 1 previous similar message [ 3417.066245] Lustre: Skipped 4 previous similar messages [ 3420.498627] Lustre: server umount lustre-OST0001 complete [ 3427.928252] Lustre: server umount lustre-MDT0000 complete [ 3429.095033] LustreError: 192949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313367 with bad export cookie 12953599281784190282 [ 3429.098070] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.099406] LustreError: 192949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3429.199295] Lustre: server umount lustre-MDT0001 complete [ 3436.576762] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:22:54 (1785313374) [ 3438.674630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.762305] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3438.774773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3438.844474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.109305] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3442.858243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.894804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3442.908833] Lustre: 199177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3444.289974] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3446.549583] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3448.273372] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3450.428452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.487654] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3452.403584] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3452.433034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3454.735172] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3456.867734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3456.895386] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3458.856115] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3460.132378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3461.183391] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3462.623793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3462.623951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.625877] LustreError: Skipped 1 previous similar message [ 3462.628932] Lustre: Skipped 2 previous similar messages [ 3462.629350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3462.633495] Lustre: Skipped 5 previous similar messages [ 3468.026642] Lustre: server umount lustre-OST0000 complete [ 3470.305854] LustreError: 201026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3470.312706] LustreError: 201026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3480.545944] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3480.547319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.548695] LustreError: Skipped 1 previous similar message [ 3480.552758] Lustre: Skipped 1 previous similar message [ 3484.622729] Lustre: server umount lustre-OST0001 complete [ 3489.247696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3493.411488] Lustre: server umount lustre-MDT0000 complete [ 3494.741602] LustreError: 199170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313433 with bad export cookie 12953599281784191444 [ 3494.746569] LustreError: 199170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3494.747946] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.889160] Lustre: server umount lustre-MDT0001 complete [ 3502.778304] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:24:01 (1785313441) [ 3508.021147] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3510.725717] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3521.631178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.709617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3521.718493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3521.753026] Lustre: lustre-MDT0000: new disk, initializing [ 3521.777363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.779781] Lustre: Skipped 3 previous similar messages [ 3521.785727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3523.004669] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3527.209891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.252162] Lustre: 207233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3527.279791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3527.282165] Lustre: Skipped 1 previous similar message [ 3527.314021] Lustre: lustre-MDT0001: new disk, initializing [ 3527.358356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3527.362408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3528.699591] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3532.391282] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3534.163669] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3536.321348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.426228] Lustre: lustre-OST0000: new disk, initializing [ 3536.428556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3536.431833] Lustre: 209197:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3538.259994] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3541.502368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3541.506822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3541.547783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3542.030278] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3543.564459] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3543.648815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3544.933683] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3544.997240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3546.592698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3546.595134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.596171] LustreError: Skipped 1 previous similar message [ 3546.600333] Lustre: Skipped 3 previous similar messages [ 3546.600830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3546.605005] Lustre: Skipped 9 previous similar messages [ 3547.671394] Lustre: server umount lustre-OST0000 complete [ 3555.034467] Lustre: server umount lustre-MDT0000 complete [ 3556.200984] LustreError: 207226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313495 with bad export cookie 12953599281784193208 [ 3556.203506] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.205668] LustreError: 207226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3556.325023] Lustre: server umount lustre-MDT0001 complete [ 3558.965830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.101923] LustreError: 211409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.108952] LustreError: 211409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3560.379306] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3563.083594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.458161] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3566.827668] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3568.531234] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3570.665535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3572.666591] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3574.949952] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3576.283321] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3584.512127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3585.811415] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3585.877147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3589.602212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.606226] LustreError: Skipped 1 previous similar message [ 3592.529447] Lustre: server umount lustre-OST0000 complete [ 3594.568284] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3594.634897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3595.954831] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3619.632716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3619.754267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.756880] Lustre: Skipped 5 previous similar messages [ 3621.754431] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3624.103480] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3624.582902] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3627.597138] Lustre: server umount lustre-OST0000 complete [ 3629.857438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.863482] Lustre: Skipped 6 previous similar messages [ 3634.969107] Lustre: server umount lustre-MDT0000 complete [ 3635.168098] LustreError: 211428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3635.172275] LustreError: 211428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3636.203866] LustreError: 211390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313575 with bad export cookie 12953599281784194076 [ 3636.206863] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3636.207723] LustreError: 211390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3636.328313] Lustre: server umount lustre-MDT0001 complete [ 3639.013991] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:26:17 (1785313577) [ 3641.104119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.587699] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3645.204832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.631683] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3648.944906] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3650.610846] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3652.653069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.564373] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3656.947127] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.288323] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3667.565384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3669.192153] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3669.268163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3671.455497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.518765] Lustre: lustre-OST0001: new disk, initializing [ 3671.520882] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3671.523424] Lustre: 220873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3673.382657] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3677.278922] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3678.725761] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3679.233516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3679.237449] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3679.265917] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3679.827577] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3681.235148] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3681.314864] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3684.320880] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.323199] LustreError: Skipped 1 previous similar message [ 3684.323509] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3684.326880] Lustre: Skipped 7 previous similar messages [ 3688.060983] Lustre: server umount lustre-OST0001 complete [ 3689.873299] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3689.942523] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3691.161544] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3691.226890] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3699.681908] LustreError: 219262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.687154] LustreError: 219262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3714.760911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.761784] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3719.437816] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3719.959751] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3720.165974] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3720.171278] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3720.171888] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3720.179784] Lustre: Skipped 1 previous similar message [ 3726.713725] Lustre: server umount lustre-OST0001 complete [ 3744.842178] Lustre: server umount lustre-OST0000 complete [ 3752.274439] Lustre: server umount lustre-MDT0000 complete [ 3753.439206] LustreError: 217423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313692 with bad export cookie 12953599281784195007 [ 3753.441548] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3753.443048] LustreError: 217423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3759.604160] Lustre: server umount lustre-MDT0001 complete [ 3767.016804] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:28:25 (1785313705) [ 3769.164048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.265060] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3769.276662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3769.347538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3769.350096] Lustre: Skipped 5 previous similar messages [ 3770.694561] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3773.473553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.502633] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3773.513452] Lustre: 226113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3774.986907] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3777.409454] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3779.262245] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3781.480907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.544252] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3783.405994] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3785.610993] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3789.242365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.272090] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3793.580132] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3794.084144] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3795.413745] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3795.479741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3796.814057] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3796.880302] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3798.661317] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e6c82e5c000.ost_server_uuid 50 [ 3799.151509] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e6c82e5c000.ost_server_uuid in FULL state after 0 sec [ 3799.479513] Lustre: Permanently deactivating lustre-OST0001 [ 3799.484715] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3799.562022] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3800.199063] Lustre: Permanently reactivating lustre-OST0001 [ 3807.212292] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.215493] Lustre: Skipped 8 previous similar messages [ 3807.289247] Lustre: server umount lustre-OST0001 complete [ 3814.614567] Lustre: server umount lustre-OST0000 complete [ 3827.679202] LustreError: 230677:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3830.239163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3830.303162] Lustre: server umount lustre-MDT0000 complete [ 3831.493997] LustreError: 226105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313770 with bad export cookie 12953599281784196281 [ 3831.497208] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3831.497612] LustreError: 226105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3831.613618] Lustre: server umount lustre-MDT0001 complete [ 3838.956730] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:29:37 (1785313777) [ 3841.501598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.586133] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3841.596779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3841.600266] Lustre: Skipped 5 previous similar messages [ 3842.905482] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3845.649738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.682383] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3845.684637] Lustre: Skipped 1 previous similar message [ 3845.694716] Lustre: 232798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3847.146511] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3849.606510] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3851.721221] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3854.135671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.219090] Lustre: Permanently deactivating lustre-OST0000 [ 3854.225188] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3856.243271] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3858.771548] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3859.950619] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3859.953516] Lustre: Skipped 1 previous similar message [ 3859.955432] LustreError: 235480:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3859.960414] LustreError: 235480:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3861.034047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3862.969847] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3865.257253] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.178970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3871.113823] Lustre: Permanently reactivating lustre-OST0000 [ 3871.119413] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3871.121399] Lustre: Skipped 2 previous similar messages [ 3877.361037] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3877.363478] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3877.368318] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3877.379469] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3877.387059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3877.858458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3877.861245] LustreError: Skipped 6 previous similar messages [ 3881.154355] Lustre: server umount lustre-OST0001 complete [ 3883.491306] LustreError: 234645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.491306] LustreError: 234643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.491330] LustreError: 234643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 3883.496582] LustreError: 234645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 3888.596381] Lustre: server umount lustre-OST0000 complete [ 3890.085439] Lustre: server umount lustre-MDT0000 complete [ 3891.350497] LustreError: 232790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313830 with bad export cookie 12953599281784198003 [ 3891.355029] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3891.355345] LustreError: 232790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3891.356307] LustreError: 235480:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3891.356347] LustreError: 235480:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3891.476698] Lustre: server umount lustre-MDT0001 complete [ 3895.038976] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 3896.127836] Key type lgssc unregistered [ 3896.265335] LNet: 237644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3896.269173] LNetError: 237644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3896.279470] LNet: Removed LNI 192.168.206.138@tcp [ 3896.610304] Key type .llcrypt unregistered [ 3896.611699] Key type ._llcrypt unregistered [ 3904.661922] Key type ._llcrypt registered [ 3904.663062] Key type .llcrypt registered [ 3904.698744] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 3905.016360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3905.053805] alg: No test for adler32 (adler32-zlib) [ 3905.889309] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 3905.967545] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 3907.543105] Key type lgssc registered [ 3907.867723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3910.859175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3910.864253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.000432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.369874] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3916.223937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.267127] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3916.278678] Lustre: 239078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3916.284812] Lustre: Permanently deactivating lustre-MDT0001 [ 3916.290493] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3916.409387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3917.789654] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3920.289329] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3920.428380] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3920.431066] LustreError: 239121:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3920.436385] LustreError: 239121:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3920.439635] LustreError: 239121:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3922.196175] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3924.446336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.563928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.389262] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3926.626570] LustreError: 241321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3926.634658] LustreError: 241321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3928.675990] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3930.882243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3930.956957] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3932.762777] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 3934.929192] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3936.230059] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3937.260598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3937.261420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3940.788132] Lustre: Permanently reactivating lustre-MDT0001 [ 3940.794479] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3940.796354] Lustre: Skipped 2 previous similar messages [ 3944.648702] Lustre: lustre-MDT0001: Client e209fe6f-99d4-430d-b277-3b2a75e4d91d (at 192.168.206.38@tcp) reconnecting [ 3946.690332] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3947.197847] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3948.835090] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3948.901524] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3950.214641] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3950.293604] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3951.155547] Lustre: Permanently deactivating lustre-MDT0001 [ 3951.162263] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3951.164849] Lustre: Skipped 2 previous similar messages [ 3958.781210] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3962.335763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3962.337930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.341720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.849239] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.851296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.853323] Lustre: Skipped 2 previous similar messages [ 3962.855716] Lustre: Skipped 1 previous similar message [ 3967.970643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3967.973234] Lustre: Skipped 4 previous similar messages [ 3968.150552] Lustre: server umount lustre-MDT0000 complete [ 3969.319138] LustreError: 239070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785313908 with bad export cookie 4496160800399323656 [ 3969.324140] LustreError: 239070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3969.324749] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3969.432500] Lustre: server umount lustre-MDT0001 complete [ 3976.695560] Lustre: server umount lustre-OST0000 complete [ 3980.228849] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:31:58 (1785313918) [ 3983.511615] Lustre: server umount lustre-OST0001 complete [ 3985.490312] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 3988.719670] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4000.336118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.425724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4000.430209] Lustre: Skipped 2 previous similar messages [ 4000.439638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4000.475664] Lustre: lustre-MDT0000: new disk, initializing [ 4000.504847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.512252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4001.867840] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4006.022532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.063746] Lustre: 247380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4006.086704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4006.088799] Lustre: Skipped 1 previous similar message [ 4006.125699] Lustre: lustre-MDT0001: new disk, initializing [ 4006.170262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4006.174056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4007.532253] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4011.188987] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.994994] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4015.150620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4015.247716] Lustre: lustre-OST0000: new disk, initializing [ 4015.249297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4015.252923] Lustre: 249348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4015.279349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.281507] Lustre: Skipped 1 previous similar message [ 4017.062780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4020.591833] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4024.823117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4024.827180] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4024.858321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4026.428782] LustreError: 248498:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4030.716189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.789623] Lustre: lustre-OST0001: new disk, initializing [ 4030.791461] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4030.794358] Lustre: 250463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4032.951146] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4036.519065] LustreError: 248498:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4036.525178] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4036.528711] LustreError: 247387:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4039.057928] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4040.697916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4040.702082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4040.722423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4046.623143] LustreError: 247387:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4047.884252] Lustre: lustre-OST0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 4050.914814] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.915375] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.917496] LustreError: Skipped 1 previous similar message [ 4050.923672] Lustre: Skipped 1 previous similar message [ 4052.995208] Lustre: lustre-OST0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 4052.998089] Lustre: Skipped 2 previous similar messages [ 4053.371751] Lustre: server umount lustre-OST0001 complete [ 4056.035995] LustreError: 250066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4056.041427] LustreError: 250066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4061.155792] LustreError: 250067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.161146] LustreError: 250067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4065.760030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.760258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.762997] LustreError: Skipped 1 previous similar message [ 4065.769555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.772786] Lustre: Skipped 1 previous similar message [ 4066.273688] LustreError: 250067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.278158] LustreError: 250067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4071.395169] LustreError: 249332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4071.400867] LustreError: 249332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4071.443482] Lustre: server umount lustre-OST0000 complete [ 4072.927752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4072.931184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.934880] Lustre: Skipped 1 previous similar message [ 4078.874035] Lustre: server umount lustre-MDT0000 complete [ 4080.082901] LustreError: 247372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314018 with bad export cookie 4496160800399326253 [ 4080.085131] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4080.086246] LustreError: 247372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4086.258490] Lustre: server umount lustre-MDT0001 complete [ 4090.075913] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4091.162631] Key type lgssc unregistered [ 4091.300442] LNet: 252492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.303423] LNetError: 252492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.311386] LNet: Removed LNI 192.168.206.138@tcp [ 4091.650851] Key type .llcrypt unregistered [ 4091.652426] Key type ._llcrypt unregistered [ 4105.059787] Key type ._llcrypt registered [ 4105.061139] Key type .llcrypt registered [ 4105.097562] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4105.397043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4105.430026] alg: No test for adler32 (adler32-zlib) [ 4106.276525] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4106.358524] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4107.935134] Key type lgssc registered [ 4108.257713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4110.668995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4110.673827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.743962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4111.751702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4111.801570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.055676] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4115.638183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.666815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4115.675060] Lustre: 254675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4115.763126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.101271] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4119.281699] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4120.962395] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4123.074498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.140898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4123.214606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.169451] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4127.567330] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4128.290903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4174.305042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4174.306200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.309378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4174.313767] Lustre: Skipped 1 previous similar message [ 4174.318745] Lustre: Skipped 1 previous similar message [ 4178.071675] Lustre: server umount lustre-OST0000 complete [ 4179.457981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4183.273590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.394298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.187470] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4187.620129] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4194.664031] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.667636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.667700] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4194.673117] Lustre: Skipped 1 previous similar message [ 4225.504816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4225.505844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.506870] LustreError: Skipped 2 previous similar messages [ 4225.513484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4225.517144] Lustre: Skipped 1 previous similar message [ 4226.717349] Lustre: server umount lustre-OST0000 complete [ 4228.575745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4228.578339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.581986] Lustre: Skipped 1 previous similar message [ 4228.583669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.624209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.624889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.744257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.746537] Lustre: Skipped 2 previous similar messages [ 4242.399106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4242.464339] Lustre: server umount lustre-MDT0000 complete [ 4243.765293] LustreError: 254666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314182 with bad export cookie 11020296009089129478 [ 4243.768721] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.771041] LustreError: 254666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4243.898015] Lustre: server umount lustre-MDT0001 complete [ 4247.725782] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4248.794066] Key type lgssc unregistered [ 4248.917435] LNet: 260090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.921195] LNetError: 260090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.928312] LNet: Removed LNI 192.168.206.138@tcp [ 4249.250323] Key type .llcrypt unregistered [ 4249.251537] Key type ._llcrypt unregistered [ 4257.787950] Key type ._llcrypt registered [ 4257.788972] Key type .llcrypt registered [ 4257.823553] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4258.221435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4258.232338] alg: No test for adler32 (adler32-zlib) [ 4259.085728] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4259.175070] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4260.751129] Key type lgssc registered [ 4261.107239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4263.581619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4263.587131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.716797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.006315] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4268.569574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.694660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.964389] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4272.277360] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4273.955886] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4276.159567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4276.271519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.185671] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4280.496216] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4290.597334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4298.719745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4298.721704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.725231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4300.770336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4300.770691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4300.776676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4300.779021] Lustre: Skipped 1 previous similar message [ 4304.400070] Lustre: server umount lustre-OST0000 complete [ 4305.888601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.889390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.893710] Lustre: Skipped 1 previous similar message [ 4311.008500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4311.011384] Lustre: Skipped 2 previous similar messages [ 4311.836880] Lustre: server umount lustre-MDT0000 complete [ 4313.128501] LustreError: 261463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314252 with bad export cookie 6741739707137359760 [ 4313.131969] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.134740] LustreError: 261463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4313.272860] Lustre: server umount lustre-MDT0001 complete [ 4316.568587] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4317.503380] Key type lgssc unregistered [ 4317.624337] LNet: 265803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4317.627585] LNetError: 265803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4317.637277] LNet: Removed LNI 192.168.206.138@tcp [ 4317.920025] Key type .llcrypt unregistered [ 4317.921017] Key type ._llcrypt unregistered [ 4324.465410] Key type ._llcrypt registered [ 4324.466363] Key type .llcrypt registered [ 4324.511893] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4324.849430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4324.881608] alg: No test for adler32 (adler32-zlib) [ 4325.763833] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4325.863370] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4327.439137] Key type lgssc registered [ 4327.807839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4330.485076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4330.491522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.626425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.897278] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4335.546255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4335.652523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.977694] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4339.296796] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4341.121435] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.287175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.391369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4345.251289] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4347.516750] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4356.707357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4362.212548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.213220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4362.218319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.221285] Lustre: Skipped 1 previous similar message [ 4363.737240] Lustre: server umount lustre-OST0000 complete [ 4366.304065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.307249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.312138] Lustre: Skipped 1 previous similar message [ 4366.314372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4367.187415] Lustre: server umount lustre-MDT0000 complete [ 4367.327953] LustreError: 267001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.328544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.333120] LustreError: 267001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4368.387171] LustreError: 266982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314307 with bad export cookie 13560674605381228256 [ 4368.390778] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4368.391984] LustreError: 266982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4368.511797] Lustre: server umount lustre-MDT0001 complete [ 4372.078848] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4373.117828] Key type lgssc unregistered [ 4373.251459] LNet: 270692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4373.253553] LNetError: 270692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4373.260421] LNet: Removed LNI 192.168.206.138@tcp [ 4373.551477] Key type .llcrypt unregistered [ 4373.553130] Key type ._llcrypt unregistered [ 4382.437752] Key type ._llcrypt registered [ 4382.438801] Key type .llcrypt registered [ 4382.474845] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4382.815728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4382.844728] alg: No test for adler32 (adler32-zlib) [ 4383.687714] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4383.770848] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4385.351120] Key type lgssc registered [ 4385.684219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4388.089713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4388.096679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.225335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4390.511506] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4393.124914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.236408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4394.448396] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4396.694963] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.420797] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.617084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.737751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.623773] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4404.855352] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4415.083966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4424.671747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4424.674874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.678606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4425.698780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4425.700345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4425.702236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.690229] Lustre: server umount lustre-OST0000 complete [ 4430.816748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4430.817569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4430.821522] Lustre: Skipped 1 previous similar message [ 4430.823233] Lustre: Skipped 1 previous similar message [ 4435.936655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4435.939149] Lustre: Skipped 2 previous similar messages [ 4436.061514] Lustre: server umount lustre-MDT0000 complete [ 4437.317027] LustreError: 274640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314376 with bad export cookie 13794059352175630818 [ 4437.319602] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.320248] LustreError: 274640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4437.434265] Lustre: server umount lustre-MDT0001 complete [ 4441.259981] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4442.352314] Key type lgssc unregistered [ 4442.483343] LNet: 276402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4442.485748] LNetError: 276402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4442.494257] LNet: Removed LNI 192.168.206.138@tcp [ 4442.798290] Key type .llcrypt unregistered [ 4442.799865] Key type ._llcrypt unregistered [ 4449.662965] Key type ._llcrypt registered [ 4449.663899] Key type .llcrypt registered [ 4449.703217] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4449.987541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.993733] alg: No test for adler32 (adler32-zlib) [ 4450.845773] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4450.941090] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4452.527160] Key type lgssc registered [ 4452.875677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4455.314348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4455.320591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.466279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.780619] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4460.458771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.582096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.968306] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4464.203563] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4465.815589] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4467.903305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4468.001699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.729403] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4472.076423] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4481.318974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4486.624389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4486.624573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.626522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4486.627349] LustreError: Skipped 1 previous similar message [ 4486.631269] Lustre: Skipped 1 previous similar message [ 4488.275203] Lustre: server umount lustre-OST0000 complete [ 4491.232295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4491.235238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.239962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4491.242319] Lustre: Skipped 1 previous similar message [ 4491.674716] Lustre: server umount lustre-MDT0000 complete [ 4491.746025] LustreError: 277595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4491.751566] LustreError: 277595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4492.936244] LustreError: 277581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314431 with bad export cookie 2117187862790409133 [ 4492.939396] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4492.940540] LustreError: 277581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4493.069291] Lustre: server umount lustre-MDT0001 complete [ 4496.568510] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4497.586334] Key type lgssc unregistered [ 4497.710386] LNet: 281327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4497.713210] LNetError: 281327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4497.724274] LNet: Removed LNI 192.168.206.138@tcp [ 4498.032653] Key type .llcrypt unregistered [ 4498.034217] Key type ._llcrypt unregistered [ 4501.501060] Key type ._llcrypt registered [ 4501.502285] Key type .llcrypt registered [ 4501.537449] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4506.970241] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4513.030454] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4513.369579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4513.420127] alg: No test for adler32 (adler32-zlib) [ 4514.275640] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4514.360286] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4515.943138] Key type lgssc registered [ 4516.308818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4527.406693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4527.412688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4528.489158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4528.497868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4528.533416] Lustre: lustre-MDT0000: new disk, initializing [ 4528.552170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.558157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4529.919800] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4533.879230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.919207] Lustre: 284852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4533.942564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4533.944184] Lustre: Skipped 1 previous similar message [ 4533.977908] Lustre: lustre-MDT0001: new disk, initializing [ 4534.008318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4534.019122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4534.023485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4535.295720] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4538.859276] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4540.555485] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4542.624050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4542.715931] Lustre: lustre-OST0000: new disk, initializing [ 4542.717495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4542.719690] Lustre: 286819:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4542.745140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.477474] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4547.580069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4547.583274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4547.612069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4548.064835] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4549.318200] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4549.381272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4550.677870] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4550.743373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.674130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.674349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.676996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.680019] Lustre: Skipped 1 previous similar message [ 4553.422677] Lustre: server umount lustre-OST0000 complete [ 4554.719893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4554.722231] LustreError: Skipped 1 previous similar message [ 4554.724143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.727637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.729895] Lustre: Skipped 1 previous similar message [ 4557.793452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4557.793774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.722537] Lustre: server umount lustre-MDT0000 complete [ 4561.903112] LustreError: 284845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314500 with bad export cookie 4589658427260694107 [ 4561.906390] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.909865] LustreError: 284845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4562.014797] Lustre: server umount lustre-MDT0001 complete [ 4564.763850] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:41:43 (1785314503) [ 4566.875811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.030540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.285093] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4570.984080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.098489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4572.376794] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4574.723701] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4576.508488] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4578.676869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4580.633787] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4582.869024] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4596.191923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4596.194545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4596.199494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4596.201379] Lustre: Skipped 1 previous similar message [ 4602.068566] Lustre: server umount lustre-OST0000 complete [ 4605.921194] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4605.923817] LustreError: Skipped 1 previous similar message [ 4605.925101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.929158] Lustre: Skipped 1 previous similar message [ 4605.931312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4605.932837] Lustre: Skipped 2 previous similar messages [ 4609.433705] Lustre: server umount lustre-MDT0000 complete [ 4610.610223] LustreError: 292031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314549 with bad export cookie 4589658427260694982 [ 4610.611441] LustreError: 289963:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4610.613108] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.614604] LustreError: 292031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4610.716499] Lustre: server umount lustre-MDT0001 complete [ 4614.042402] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4615.000263] Key type lgssc unregistered [ 4615.113323] LNet: 292781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4615.115664] LNetError: 292781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4615.128264] LNet: Removed LNI 192.168.206.138@tcp [ 4615.413464] Key type .llcrypt unregistered [ 4615.414470] Key type ._llcrypt unregistered [ 4623.481453] Key type ._llcrypt registered [ 4623.483078] Key type .llcrypt registered [ 4623.518126] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4629.262365] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4629.563439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4629.574355] alg: No test for adler32 (adler32-zlib) [ 4630.421633] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4630.501741] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4632.079136] Key type lgssc registered [ 4632.416961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4646.286115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4646.290992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.369650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4647.381381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4647.422116] Lustre: lustre-MDT0000: new disk, initializing [ 4647.446048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.454374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4648.809242] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4653.074317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.117795] Lustre: 296578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.148188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4653.150305] Lustre: Skipped 1 previous similar message [ 4653.189911] Lustre: lustre-MDT0001: new disk, initializing [ 4653.228030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.241660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.246510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4654.623478] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4658.257432] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4660.010171] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.177444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4662.275711] Lustre: lustre-OST2710: new disk, initializing [ 4662.277493] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4662.280071] Lustre: 298540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4662.308287] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4663.544095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4663.546895] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4663.584286] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4664.264143] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4667.916376] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.042289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4670.096883] Lustre: lustre-OST03e8: new disk, initializing [ 4670.098448] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4670.101160] Lustre: 299604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4670.130448] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4671.880606] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4674.158685] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4676.790615] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4676.857585] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4678.244934] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4678.321303] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4679.682360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4679.685772] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4679.715827] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4679.716842] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4679.782781] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4681.046846] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4681.109477] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4684.255993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.258358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.262621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.771317] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.773378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.775985] Lustre: Skipped 2 previous similar messages [ 4684.780188] Lustre: Skipped 2 previous similar messages [ 4688.216553] Lustre: server umount lustre-MDT0000 complete [ 4689.358983] LustreError: 296569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314628 with bad export cookie 4688709386556206942 [ 4689.362034] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4689.362425] LustreError: 296569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4689.467343] Lustre: server umount lustre-MDT0001 complete [ 4700.938624] Lustre: server umount lustre-OST2710 complete [ 4712.471123] Lustre: server umount lustre-OST03e8 complete [ 4718.075161] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 4721.447496] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4733.049418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4733.127824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4733.137695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4733.175310] Lustre: lustre-MDT0000: new disk, initializing [ 4733.194989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.219704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4734.587521] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4738.453199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.490407] Lustre: 304843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4738.518855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4738.587361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4739.775013] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4743.399740] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4745.061461] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.108893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.203757] Lustre: lustre-OST0000: new disk, initializing [ 4747.205163] Lustre: Skipped 1 previous similar message [ 4747.206677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4747.207563] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4747.208595] Lustre: Skipped 2 previous similar messages [ 4747.213347] Lustre: 306811:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4747.243339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4747.245384] Lustre: Skipped 1 previous similar message [ 4749.019594] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4752.360550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4752.363220] Lustre: Skipped 1 previous similar message [ 4752.364928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4752.376930] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4752.645642] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4754.083835] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4754.159521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4755.682964] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4755.762387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4757.473595] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.474590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.476234] LustreError: Skipped 1 previous similar message [ 4757.480348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.482721] Lustre: Skipped 1 previous similar message [ 4758.549209] Lustre: server umount lustre-OST0000 complete [ 4762.592671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.597315] Lustre: Skipped 2 previous similar messages [ 4767.712269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4767.715329] Lustre: Skipped 3 previous similar messages [ 4774.367122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4774.430312] Lustre: server umount lustre-MDT0000 complete [ 4775.664236] LustreError: 305635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314714 with bad export cookie 4688709386556208601 [ 4775.666971] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.669524] LustreError: 305635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4775.784543] Lustre: server umount lustre-MDT0001 complete [ 4778.514364] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:45:16 (1785314716) [ 4779.057624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4779.680971] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:45:17 (1785314717) [ 4780.718567] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4789.514362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.598560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4789.606107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4789.662500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4791.043502] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4793.835415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.872073] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4793.883343] Lustre: 310674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4795.320902] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4811.105555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.170460] Lustre: Denying initial registration attempt from nid 192.168.206.138@tcp, specified as failover [ 4811.173780] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4811.177311] LustreError: 312126:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4811.181210] LustreError: 312126:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4811.184039] LustreError: 312126:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4811.206718] Lustre: server umount lustre-OST0000 complete [ 4811.208206] LustreError: 312126:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4814.815690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4814.818405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.822758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.824911] Lustre: Skipped 2 previous similar messages [ 4819.414503] Lustre: server umount lustre-MDT0000 complete [ 4820.618092] LustreError: 310665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314759 with bad export cookie 4688709386556209483 [ 4820.621120] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.622266] LustreError: 310665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4820.731732] Lustre: server umount lustre-MDT0001 complete [ 4824.308023] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4825.339803] Key type lgssc unregistered [ 4825.466503] LNet: 313246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4825.470136] LNetError: 313246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4825.476450] LNet: Removed LNI 192.168.206.138@tcp [ 4825.785066] Key type .llcrypt unregistered [ 4825.786228] Key type ._llcrypt unregistered [ 4829.510640] Key type ._llcrypt registered [ 4829.511942] Key type .llcrypt registered [ 4829.550629] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4830.010626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4830.031914] alg: No test for adler32 (adler32-zlib) [ 4830.892981] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4830.981610] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4832.559122] Key type lgssc registered [ 4832.891781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4856.804589] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4859.588063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4859.592625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.667139] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4860.673584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4860.721061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4861.957601] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4864.631829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.664537] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4864.676117] Lustre: 315982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4864.760871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4866.021629] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4868.361185] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4870.093254] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4885.218046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4885.282908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4885.340097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.138839] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4889.475323] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4890.158353] Lustre: server umount lustre-OST0000 complete [ 4892.657409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.658207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.662615] Lustre: Skipped 1 previous similar message [ 4897.423753] Lustre: server umount lustre-MDT0000 complete [ 4898.564035] LustreError: 316726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314837 with bad export cookie 1131294551566783859 [ 4898.567025] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4898.568197] LustreError: 316726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4898.672443] Lustre: server umount lustre-MDT0001 complete [ 4902.089812] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 4903.016894] Key type lgssc unregistered [ 4903.137328] LNet: 319546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4903.141254] LNetError: 319546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4903.151290] LNet: Removed LNI 192.168.206.138@tcp [ 4903.469482] Key type .llcrypt unregistered [ 4903.470508] Key type ._llcrypt unregistered [ 4912.167979] Key type ._llcrypt registered [ 4912.169293] Key type .llcrypt registered [ 4912.201736] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 4912.611147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4912.620203] alg: No test for adler32 (adler32-zlib) [ 4913.460549] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 4913.534232] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 4915.103173] Key type lgssc registered [ 4915.451160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4917.989738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4917.997406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.135070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.426017] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4923.043056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.154443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.465981] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4926.616747] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4928.550406] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4932.900049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.013728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.002168] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4937.409980] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4953.098838] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 4954.080072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4954.083170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.088053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4954.593902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.597622] Lustre: Skipped 1 previous similar message [ 4956.185051] Lustre: server umount lustre-MDT0000 complete [ 4957.443227] LustreError: 320963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314896 with bad export cookie 16923123808026297282 [ 4957.445744] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4957.448405] LustreError: 320963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4958.210195] LustreError: 322389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.214659] LustreError: 322389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4958.217422] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 4958.219187] Lustre: Skipped 2 previous similar messages [ 4959.712029] LustreError: 320935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.712241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.719732] LustreError: 322939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.719961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4962.728775] Lustre: Skipped 1 previous similar message [ 4963.626441] Lustre: server umount lustre-MDT0001 complete [ 4965.871971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4968.775941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.239748] LustreError: 320249:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ce06ce5a x1872038306276224/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4982.358301] LustreError: 322768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4982.364204] LustreError: 322768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4982.384359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4982.406962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4983.870738] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4986.534558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.662485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4987.683458] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4987.921611] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 4990.202103] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4992.805255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4994.808449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4996.606866] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4997.113445] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5002.208255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.212687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.219236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.221259] Lustre: Skipped 1 previous similar message [ 5003.235682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5004.504591] Lustre: server umount lustre-OST0000 complete [ 5007.327954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5007.330639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.334653] Lustre: Skipped 1 previous similar message [ 5013.471912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5013.473638] Lustre: Skipped 6 previous similar messages [ 5020.127141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5020.188934] Lustre: server umount lustre-MDT0000 complete [ 5021.400285] LustreError: 320919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785314960 with bad export cookie 16923123808026304436 [ 5021.404550] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5021.517536] Lustre: server umount lustre-MDT0001 complete [ 5025.003504] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5026.022739] Key type lgssc unregistered [ 5026.151322] LNet: 327221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5026.154133] LNetError: 327221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5026.162297] LNet: Removed LNI 192.168.206.138@tcp [ 5026.499931] Key type .llcrypt unregistered [ 5026.501676] Key type ._llcrypt unregistered [ 5035.758250] Key type ._llcrypt registered [ 5035.760086] Key type .llcrypt registered [ 5035.797743] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5036.194851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5036.204041] alg: No test for adler32 (adler32-zlib) [ 5037.068203] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5037.153720] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5038.727100] Key type lgssc registered [ 5039.081223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.591250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.598141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.738345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.089836] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5047.011693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.164240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5048.518862] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5050.937904] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5052.831604] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5057.505153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.505477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5057.505961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.511582] Lustre: Skipped 1 previous similar message [ 5058.528301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5058.530433] Lustre: Skipped 2 previous similar messages [ 5063.648058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5063.649951] Lustre: Skipped 1 previous similar message [ 5068.255106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5068.319502] Lustre: server umount lustre-MDT0000 complete [ 5068.768073] LustreError: 328613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.773342] LustreError: 328613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5069.555389] LustreError: 328594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315008 with bad export cookie 5829004215047539047 [ 5069.559667] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.559966] LustreError: 328594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5069.695468] Lustre: server umount lustre-MDT0001 complete [ 5074.661413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.745942] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5074.756793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5074.821698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5076.039060] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5078.702413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.732435] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5078.742051] Lustre: 331197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5078.827262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.115387] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5082.379381] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.259353] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5088.802315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.869402] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5088.875223] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5088.877947] LustreError: 333079:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5088.882393] LustreError: 333079:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5088.884804] LustreError: 333079:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5088.907768] Lustre: server umount lustre-OST0000 complete [ 5088.910051] LustreError: 333079:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5093.794311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5093.861302] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5093.932193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.909533] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5098.352622] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5099.531489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5100.744769] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5100.774866] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5100.803228] Lustre: lustre-OST0001: new disk, initializing [ 5100.805511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5100.807856] Lustre: 334618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5102.574582] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5104.989858] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5106.001463] Lustre: server umount lustre-OST0001 complete [ 5106.172180] LustreError: 333547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.728449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.728877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.732271] LustreError: Skipped 1 previous similar message [ 5109.737658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.742148] Lustre: Skipped 1 previous similar message [ 5111.267502] LustreError: 333547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5111.272893] LustreError: 333547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5113.360115] Lustre: server umount lustre-OST0000 complete [ 5114.847859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5114.850145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.853704] Lustre: Skipped 1 previous similar message [ 5114.855510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.858324] Lustre: Skipped 2 previous similar messages [ 5120.734272] Lustre: server umount lustre-MDT0000 complete [ 5121.503791] LustreError: 331951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.509587] LustreError: 331951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5121.998991] LustreError: 331234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315060 with bad export cookie 5829004215047539586 [ 5122.002201] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.002458] LustreError: 331234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5122.107852] Lustre: server umount lustre-MDT0001 complete [ 5125.477739] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5126.436143] Key type lgssc unregistered [ 5126.560422] LNet: 336570:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.563451] LNetError: 336570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.570248] LNet: Removed LNI 192.168.206.138@tcp [ 5126.836208] Key type .llcrypt unregistered [ 5126.837123] Key type ._llcrypt unregistered [ 5156.121318] Key type ._llcrypt registered [ 5156.122232] Key type .llcrypt registered [ 5156.160444] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5162.286847] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5162.763280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5162.779609] alg: No test for adler32 (adler32-zlib) [ 5163.636115] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5163.732653] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5165.319155] Key type lgssc registered [ 5165.710285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5177.211917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5177.221994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.326120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5178.339942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5178.380420] Lustre: lustre-MDT0000: new disk, initializing [ 5178.409589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5178.417990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5179.750946] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5183.877428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.916067] Lustre: 341472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5183.937361] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5183.939612] Lustre: Skipped 1 previous similar message [ 5183.974023] Lustre: lustre-MDT0001: new disk, initializing [ 5184.006955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5184.020661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5184.024599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5185.339990] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5188.853515] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5190.611153] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5192.640552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.753148] Lustre: lustre-OST0000: new disk, initializing [ 5192.754535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5192.756776] Lustre: 343440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5192.781021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.531960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5193.535915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5193.574513] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5194.530405] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5198.328482] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5199.796309] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5199.878495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5201.315356] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5201.398230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5203.940141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5203.941318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5203.942126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5203.944628] Lustre: Skipped 1 previous similar message [ 5203.948413] LustreError: Skipped 1 previous similar message [ 5208.144696] Lustre: server umount lustre-OST0000 complete [ 5209.567706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5209.570547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.574677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.576579] Lustre: Skipped 1 previous similar message [ 5214.176660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5214.177065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.517697] Lustre: server umount lustre-MDT0000 complete [ 5216.821140] LustreError: 341463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315155 with bad export cookie 15815523124639528615 [ 5216.823967] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5216.825029] LustreError: 341463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5216.954300] Lustre: server umount lustre-MDT0001 complete [ 5219.779462] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:52:38 (1785315158) [ 5222.193443] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:52:40 (1785315160) [ 5224.844418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.002732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.331574] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5229.033072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.137860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5230.420205] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5232.602382] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.306226] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.422589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5238.292179] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5240.632429] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5254.398110] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5254.947588] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5255.499118] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5256.058736] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5256.674657] Lustre: Failing over lustre-MDT0000 [ 5256.835519] Lustre: server umount lustre-MDT0000 complete [ 5259.743877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5259.747404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.753339] LustreError: 346192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.760287] LustreError: 346192:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5261.280562] LustreError: 346191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.285227] LustreError: 346191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5266.400375] LustreError: 346210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.407401] LustreError: 346210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5269.618362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.658133] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.740677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.743385] Lustre: Skipped 1 previous similar message [ 5269.759640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5271.034860] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5271.554770] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5275.110853] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5275.120298] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5275.138209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5276.875076] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5277.410134] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5279.418348] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5280.142717] Lustre: Failing over lustre-MDT0000 [ 5280.224390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5280.225938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5280.230097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5280.236427] Lustre: Skipped 3 previous similar messages [ 5280.240097] Lustre: Skipped 1 previous similar message [ 5280.372610] Lustre: server umount lustre-MDT0000 complete [ 5281.804833] LustreError: 348680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.813642] LustreError: 348680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5286.914670] LustreError: 346187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.922379] LustreError: 346187:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5293.527128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.571132] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5293.661371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.686251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5295.082813] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5297.154050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5299.175828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5299.178515] Lustre: Skipped 2 previous similar messages [ 5299.189135] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5299.205540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5301.094250] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5301.674657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5304.288610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5304.289247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.289893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5304.289897] Lustre: Skipped 2 previous similar messages [ 5304.303282] Lustre: Skipped 2 previous similar messages [ 5310.101632] Lustre: server umount lustre-MDT0000 complete [ 5311.302320] LustreError: 346173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315250 with bad export cookie 15815523124639531296 [ 5311.305099] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5311.306785] LustreError: 346173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5311.419974] Lustre: server umount lustre-MDT0001 complete [ 5322.906331] Lustre: server umount lustre-OST0000 complete [ 5328.234047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.376505] LustreError: 352542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.382854] LustreError: 352542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5328.400599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.746107] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5332.840077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.428652] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5337.049876] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5339.121231] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5341.332478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5343.248605] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5345.607529] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5353.764138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5358.047958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.051662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.057559] Lustre: Skipped 1 previous similar message [ 5358.059638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.061974] Lustre: Skipped 5 previous similar messages [ 5363.337484] Lustre: server umount lustre-OST0000 complete [ 5370.651614] Lustre: server umount lustre-MDT0000 complete [ 5371.924202] LustreError: 354384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315310 with bad export cookie 15815523124639532017 [ 5371.927933] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5371.928200] LustreError: 354384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5372.046032] Lustre: server umount lustre-MDT0001 complete [ 5375.638397] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5376.637553] Key type lgssc unregistered [ 5376.762289] LNet: 356091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5376.764653] LNetError: 356091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5376.774215] LNet: Removed LNI 192.168.206.138@tcp [ 5377.032834] Key type .llcrypt unregistered [ 5377.033823] Key type ._llcrypt unregistered [ 5386.723426] Key type ._llcrypt registered [ 5386.724429] Key type .llcrypt registered [ 5386.755471] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5387.034819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5387.065399] alg: No test for adler32 (adler32-zlib) [ 5387.915393] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5388.006922] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5389.583119] Key type lgssc registered [ 5389.933802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5392.474746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5392.481030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.617292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.934195] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5397.624316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.741065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5399.099933] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5401.453724] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5403.205675] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5405.328298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5405.437491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5407.242715] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5409.484978] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5419.816724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5425.120620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.122275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5425.124682] Lustre: Skipped 1 previous similar message [ 5425.127617] Lustre: Skipped 2 previous similar messages [ 5429.906159] Lustre: server umount lustre-MDT0000 complete [ 5430.239910] LustreError: 357582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5430.247173] LustreError: 357582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5431.057551] LustreError: 357563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315369 with bad export cookie 4265338724120737591 [ 5431.060614] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.062833] LustreError: 357563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5431.177791] Lustre: server umount lustre-MDT0001 complete [ 5442.648808] Lustre: server umount lustre-OST0000 complete [ 5446.957818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5447.105031] LustreError: 361312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.128421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5448.484548] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5451.356743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.468487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5452.769982] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5455.227728] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5457.071886] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5459.366917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5461.246835] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5463.580286] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5471.778353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5481.953139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.956041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5481.956995] Lustre: Skipped 2 previous similar messages [ 5487.072754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.074989] Lustre: Skipped 4 previous similar messages [ 5487.250652] Lustre: server umount lustre-MDT0000 complete [ 5488.454903] LustreError: 363925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315427 with bad export cookie 4265338724120738690 [ 5488.458109] LustreError: 363925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5488.458153] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5488.558970] Lustre: server umount lustre-MDT0001 complete [ 5500.132950] Lustre: server umount lustre-OST0000 complete [ 5505.741109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.901622] LustreError: 365239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.930104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.932938] Lustre: Skipped 1 previous similar message [ 5507.323443] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5510.221044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.755190] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5514.152884] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5515.960712] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5518.170343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5520.166455] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5522.500437] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5530.602248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5535.715781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.715861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.715880] Lustre: Skipped 1 previous similar message [ 5535.717132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.719100] LustreError: Skipped 1 previous similar message [ 5536.397866] Lustre: server umount lustre-OST0000 complete [ 5539.736732] Lustre: server umount lustre-MDT0000 complete [ 5540.833440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.834346] LustreError: 365234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.840283] Lustre: Skipped 1 previous similar message [ 5541.119182] LustreError: 367083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315479 with bad export cookie 4265338724120739698 [ 5541.123058] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5541.126872] LustreError: 367083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5541.255287] Lustre: server umount lustre-MDT0001 complete [ 5545.052731] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5546.105700] Key type lgssc unregistered [ 5546.232327] LNet: 368791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5546.236156] LNetError: 368791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5546.242284] LNet: Removed LNI 192.168.206.138@tcp [ 5546.538913] Key type .llcrypt unregistered [ 5546.540303] Key type ._llcrypt unregistered [ 5555.737328] Key type ._llcrypt registered [ 5555.738949] Key type .llcrypt registered [ 5555.777212] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5556.108158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5556.148371] alg: No test for adler32 (adler32-zlib) [ 5557.000586] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5557.085503] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5558.671102] Key type lgssc registered [ 5559.036883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5561.618023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5561.626460] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.630156] LustreError: 370209:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5561.639208] LustreError: 370209:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5561.641505] LustreError: 370209:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5561.643853] LustreError: 370209:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5561.647373] LustreError: 370209:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5564.114040] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5564.117745] LustreError: 370605:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5564.125794] LustreError: 370605:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5564.127795] LustreError: 370605:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5564.130267] LustreError: 370605:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5564.132591] LustreError: 370605:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5570.246842] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5571.233541] Key type lgssc unregistered [ 5571.366238] LNet: 371605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5571.368700] LNetError: 371605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5571.376315] LNet: Removed LNI 192.168.206.138@tcp [ 5571.673334] Key type .llcrypt unregistered [ 5571.674344] Key type ._llcrypt unregistered [ 5577.458547] Key type ._llcrypt registered [ 5577.459427] Key type .llcrypt registered [ 5577.496373] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5583.689307] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5583.988209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5584.023767] alg: No test for adler32 (adler32-zlib) [ 5584.889682] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5584.993515] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5586.575093] Key type lgssc registered [ 5586.930654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5598.209764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5598.229514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.309152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.319265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5599.355617] Lustre: lustre-MDT0000: new disk, initializing [ 5599.375587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5599.381411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5600.771714] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5604.912450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.954693] Lustre: 374734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5604.979195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5604.980890] Lustre: Skipped 1 previous similar message [ 5605.015962] Lustre: lustre-MDT0001: new disk, initializing [ 5605.047229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5605.061210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5605.066405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5606.402209] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5609.979544] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5611.813451] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5614.052156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5614.143503] Lustre: lustre-OST0000: new disk, initializing [ 5614.145370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5614.147883] Lustre: 376701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5614.175088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.150019] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5616.384591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5616.389481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5616.426941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5619.806112] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.234369] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5621.306498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5622.704021] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5622.772571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5626.851252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5626.851266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.855174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5626.859833] Lustre: Skipped 1 previous similar message [ 5629.522882] Lustre: server umount lustre-OST0000 complete [ 5630.943866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5630.946425] LustreError: Skipped 1 previous similar message [ 5630.947900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.952505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.955410] Lustre: Skipped 1 previous similar message [ 5631.969066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.969113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5631.979055] Lustre: Skipped 1 previous similar message [ 5637.088992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.092897] Lustre: Skipped 1 previous similar message [ 5642.208244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5642.211039] Lustre: Skipped 1 previous similar message [ 5645.279105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.349970] Lustre: server umount lustre-MDT0000 complete [ 5646.546048] LustreError: 374727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315585 with bad export cookie 4972439149731672614 [ 5646.548783] LustreError: 374727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5646.548882] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5646.655900] Lustre: server umount lustre-MDT0001 complete [ 5649.226424] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5649.720224] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:59:48 (1785315588) [ 5651.844266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.989342] LustreError: 379158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.996110] LustreError: 379158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5652.009964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.358678] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5656.180056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.309052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5657.781842] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5660.055328] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.821605] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5663.922028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.770269] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5668.076838] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.265895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5670.320474] Lustre: lustre-OST0001: new disk, initializing [ 5670.322326] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5670.325163] Lustre: 381991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5670.352868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.356243] Lustre: Skipped 1 previous similar message [ 5672.134032] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5673.215144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5673.218573] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5673.244239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5675.881282] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5678.560144] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5678.560219] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.562355] LustreError: Skipped 1 previous similar message [ 5678.562888] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5678.566293] Lustre: Skipped 1 previous similar message [ 5678.570610] Lustre: Skipped 1 previous similar message [ 5683.899909] Lustre: server umount lustre-OST0001 complete [ 5686.756826] LustreError: 380981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.761343] LustreError: 380981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5691.878510] LustreError: 382458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.883227] LustreError: 382458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5696.994773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.994796] LustreError: 380982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.995301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5696.999507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5697.003682] LustreError: 380982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5697.009865] Lustre: Skipped 4 previous similar messages [ 5702.118541] LustreError: 380982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.122839] LustreError: 380982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5710.303101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5710.359228] Lustre: server umount lustre-OST0000 complete [ 5712.352826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.356843] Lustre: Skipped 2 previous similar messages [ 5717.853916] Lustre: server umount lustre-MDT0000 complete [ 5719.109346] LustreError: 379141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315657 with bad export cookie 4972439149731673482 [ 5719.112109] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5719.112764] LustreError: 379141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5725.303313] Lustre: server umount lustre-MDT0001 complete [ 5729.171903] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5730.173022] Key type lgssc unregistered [ 5730.299406] LNet: 383990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.301957] LNetError: 383990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5730.311281] LNet: Removed LNI 192.168.206.138@tcp [ 5730.637071] Key type .llcrypt unregistered [ 5730.638762] Key type ._llcrypt unregistered [ 5748.370591] Key type ._llcrypt registered [ 5748.372378] Key type .llcrypt registered [ 5748.420105] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5748.840067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5748.906627] alg: No test for adler32 (adler32-zlib) [ 5749.767773] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5749.862791] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5751.439143] Key type lgssc registered [ 5751.805438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5754.355027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5754.363179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.445860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5755.453818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5755.512881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.896330] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5759.753306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.784104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5759.795159] Lustre: 386765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5759.873927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5761.189468] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5763.725097] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5765.687098] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.696956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.698148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.701484] Lustre: Skipped 1 previous similar message [ 5772.561287] Lustre: server umount lustre-MDT0000 complete [ 5773.760578] LustreError: 387505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315712 with bad export cookie 1140467209541122549 [ 5773.763737] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.764536] LustreError: 387505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5773.884836] Lustre: server umount lustre-MDT0001 complete [ 5777.012294] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:01:55 (1785315715) [ 5779.287787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.458066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5780.864295] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5783.737104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.857894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5785.244120] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5787.685478] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5789.527256] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5791.748925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5791.814849] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5791.872721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.852446] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5796.236258] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5799.924494] Lustre: Permanently deactivating lustre-OST0000 [ 5799.929696] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5804.531916] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5805.700482] LustreError: 392258:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5805.706226] LustreError: 392258:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5805.708774] LustreError: 392258:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5806.307645] LustreError: 392306:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5806.310337] LustreError: 392306:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5813.008892] Lustre: server umount lustre-OST0000 complete [ 5813.883802] LustreError: 392556:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [ 5813.886208] LustreError: 392556:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5814.751852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5814.753760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.757273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5814.759220] Lustre: Skipped 1 previous similar message [ 5820.384374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.386644] Lustre: Skipped 3 previous similar messages [ 5824.505899] Lustre: server umount lustre-MDT0000 complete [ 5825.503926] LustreError: 389292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5825.509444] LustreError: 389292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5825.704991] LustreError: 389254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315764 with bad export cookie 1140467209541123116 [ 5825.708198] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5825.708430] LustreError: 389254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5835.278811] Lustre: server umount lustre-MDT0001 complete [ 5838.190425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.489183] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5840.340894] LustreError: 393714:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5842.399461] Lustre: 394054:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.138@tcp [ 5843.076906] Lustre: server umount MGS complete [ 5846.918598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.094722] LustreError: 394777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.100757] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5847.103956] Lustre: Skipped 1 previous similar message [ 5847.127121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5848.507117] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5851.347701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.453204] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5852.905651] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5855.307504] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5857.200710] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5857.846607] Lustre: Permanently reactivating lustre-OST0000 [ 5857.852153] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5857.854206] Lustre: Skipped 2 previous similar messages [ 5884.895133] LustreError: 396611:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5887.496309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5887.616086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.618784] Lustre: Skipped 1 previous similar message [ 5889.551663] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5891.841499] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5905.376917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5905.377102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5905.380811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5905.384493] Lustre: Skipped 2 previous similar messages [ 5909.780737] Lustre: server umount lustre-OST0000 complete [ 5913.055947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5913.058230] LustreError: Skipped 1 previous similar message [ 5913.059626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.063889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5913.065913] Lustre: Skipped 1 previous similar message [ 5925.343138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5925.410593] Lustre: server umount lustre-MDT0000 complete [ 5925.856882] LustreError: 394778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.863868] LustreError: 394778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5926.679182] LustreError: 396350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315865 with bad export cookie 1140467209541124152 [ 5926.682442] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5926.682932] LustreError: 396350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5926.798863] Lustre: server umount lustre-MDT0001 complete [ 5930.582517] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 5931.711140] Key type lgssc unregistered [ 5931.854415] LNet: 399496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5931.857093] LNetError: 399496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5931.865421] LNet: Removed LNI 192.168.206.138@tcp [ 5932.205629] Key type .llcrypt unregistered [ 5932.207506] Key type ._llcrypt unregistered [ 5938.465256] Key type ._llcrypt registered [ 5938.466247] Key type .llcrypt registered [ 5938.498926] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 5944.854459] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 5945.236538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.245342] alg: No test for adler32 (adler32-zlib) [ 5946.080124] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 5946.165372] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 5947.735119] Key type lgssc registered [ 5948.037929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5958.748922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5958.753627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.830806] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5959.841409] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5959.872740] Lustre: lustre-MDT0000: new disk, initializing [ 5959.891638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5959.898630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5961.272122] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5965.275587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.321983] Lustre: 402624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5965.346926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5965.349918] Lustre: Skipped 1 previous similar message [ 5965.384216] Lustre: lustre-MDT0001: new disk, initializing [ 5965.412245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5965.422532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5965.427299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5966.616964] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5970.347964] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5972.214858] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5974.546207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5974.660462] Lustre: lustre-OST0000: new disk, initializing [ 5974.662136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5974.665741] Lustre: 404596:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5974.695831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5976.558143] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 5977.078669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5977.082466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5977.115259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5980.176637] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5981.523496] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5981.589763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5982.902497] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5982.973887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5987.297630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.297810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.300544] LustreError: Skipped 1 previous similar message [ 5987.304335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.307034] Lustre: Skipped 1 previous similar message [ 5989.706029] Lustre: server umount lustre-OST0000 complete [ 5990.879680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5990.881974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.885822] Lustre: Skipped 1 previous similar message [ 5990.887593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.416163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5992.416831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.536125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5997.538339] Lustre: Skipped 2 previous similar messages [ 6002.656497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6002.658609] Lustre: Skipped 1 previous similar message [ 6005.215126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6005.280913] Lustre: server umount lustre-MDT0000 complete [ 6006.523093] LustreError: 402617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315945 with bad export cookie 6645243506227470291 [ 6006.526301] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6006.527563] LustreError: 402617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6006.646500] Lustre: server umount lustre-MDT0001 complete [ 6009.335868] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6009.865599] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:05:48 (1785315948) [ 6011.957365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6012.105359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.316663] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6015.965323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.389328] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6019.675536] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6021.392411] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.454127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6023.553172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.555869] Lustre: Skipped 1 previous similar message [ 6025.367996] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6027.655625] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6028.300665] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6039.127058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6041.055858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6041.058355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.062268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.796693] Lustre: server umount lustre-OST0000 complete [ 6051.807761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6051.810610] LustreError: Skipped 1 previous similar message [ 6051.812113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6051.816472] Lustre: Skipped 1 previous similar message [ 6054.096746] Lustre: server umount lustre-MDT0000 complete [ 6055.294132] LustreError: 407032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315994 with bad export cookie 6645243506227471159 [ 6055.296529] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.298440] LustreError: 407032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6055.407465] Lustre: server umount lustre-MDT0001 complete [ 6058.799521] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6059.749556] Key type lgssc unregistered [ 6059.864339] LNet: 410792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6059.867339] LNetError: 410792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6059.876432] LNet: Removed LNI 192.168.206.138@tcp [ 6060.157896] Key type .llcrypt unregistered [ 6060.159301] Key type ._llcrypt unregistered [ 6067.157454] Key type ._llcrypt registered [ 6067.158903] Key type .llcrypt registered [ 6067.192038] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6067.281223] Key type .llcrypt unregistered [ 6067.282274] Key type ._llcrypt unregistered [ 6074.048543] Key type ._llcrypt registered [ 6074.049906] Key type .llcrypt registered [ 6074.087059] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6074.394365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6074.467166] alg: No test for adler32 (adler32-zlib) [ 6075.308939] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6075.394394] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6076.967134] Key type lgssc registered [ 6077.289286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6079.866653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6079.872954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.019487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.427910] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6085.244845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6085.361776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6087.326455] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6089.803881] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6090.467064] LustreError: 412979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.472863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6090.474767] LustreError: 412979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6092.124639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.240880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6093.530396] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6100.959824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6100.962093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.965840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.498371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6102.499092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6102.503234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.635678] Lustre: server umount lustre-OST0000 complete [ 6111.201330] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6111.203913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.208158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6111.209940] Lustre: Skipped 1 previous similar message [ 6114.011365] Lustre: server umount lustre-MDT0000 complete [ 6115.300489] LustreError: 414721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316054 with bad export cookie 2794634607604412640 [ 6115.303403] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6115.306826] LustreError: 414721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6115.418418] Lustre: server umount lustre-MDT0001 complete [ 6118.885230] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6119.888474] Key type lgssc unregistered [ 6120.019410] LNet: 416126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6120.021946] LNetError: 416126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6120.033336] LNet: Removed LNI 192.168.206.138@tcp [ 6120.355947] Key type .llcrypt unregistered [ 6120.357186] Key type ._llcrypt unregistered [ 6129.144955] Key type ._llcrypt registered [ 6129.146553] Key type .llcrypt registered [ 6129.186950] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6129.584333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6129.590567] alg: No test for adler32 (adler32-zlib) [ 6130.419790] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6130.495656] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6132.063098] Key type lgssc registered [ 6132.445606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6135.088699] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6135.095919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.559343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6155.679271] LustreError: 417492:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.138@tcp: failed processing log, type 4: rc = -110 [ 6182.303150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6184.296601] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6186.799061] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6189.058053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.243366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.270479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6190.612328] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6193.463185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.595070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6194.984951] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6197.371589] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6199.333554] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6201.826061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.827097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.830050] Lustre: Skipped 1 previous similar message [ 6206.945675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6206.949022] Lustre: Skipped 2 previous similar messages [ 6207.696044] Lustre: server umount lustre-OST0000 complete [ 6208.991793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6208.993945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6208.997709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.064051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6212.065027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.127451] Lustre: server umount lustre-MDT0000 complete [ 6216.345249] LustreError: 418556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316155 with bad export cookie 16830495472820415204 [ 6216.348562] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6216.349942] LustreError: 418556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6216.480681] Lustre: server umount lustre-MDT0001 complete [ 6220.281661] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6221.315944] Key type lgssc unregistered [ 6221.446376] LNet: 421065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6221.448819] LNetError: 421065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6221.455321] LNet: Removed LNI 192.168.206.138@tcp [ 6221.761164] Key type .llcrypt unregistered [ 6221.762479] Key type ._llcrypt unregistered [ 6231.111677] Key type ._llcrypt registered [ 6231.112784] Key type .llcrypt registered [ 6231.155051] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6231.641591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6231.653898] alg: No test for adler32 (adler32-zlib) [ 6232.520096] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6232.613491] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6234.199125] Key type lgssc registered [ 6234.573052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.185444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.192621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.341072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6239.643954] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6242.464030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.593973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6243.924758] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6246.385069] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6248.141527] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.407774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6250.518869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6252.389512] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6254.818129] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.845368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6268.384072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6268.386872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.391389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.897656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.898365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.902539] Lustre: Skipped 1 previous similar message [ 6270.473782] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.38@tcp (stopping) [ 6270.475823] Lustre: Skipped 1 previous similar message [ 6272.404117] Lustre: server umount lustre-MDT0000 complete [ 6273.167518] LustreError: 422438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.38@tcp arrived at 1785316212 with bad export cookie 2690079535897289615 [ 6273.173418] LustreError: 422438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6273.853146] LustreError: 422439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316212 with bad export cookie 2690079535897288992 [ 6273.857109] LustreError: 422439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6273.857263] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6274.015996] LustreError: 423198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.016639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6274.021018] LustreError: 423198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6274.029520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.047877] LustreError: 422456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.048269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.053511] LustreError: 422456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6279.954854] Lustre: server umount lustre-OST0000 complete [ 6282.330105] Lustre: server umount lustre-MDT0001 complete [ 6286.139847] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6287.165630] Key type lgssc unregistered [ 6287.296487] LNet: 426162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6287.300856] LNetError: 426162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6287.309424] LNet: Removed LNI 192.168.206.138@tcp [ 6287.663260] Key type .llcrypt unregistered [ 6287.664575] Key type ._llcrypt unregistered [ 6296.955470] Key type ._llcrypt registered [ 6296.957264] Key type .llcrypt registered [ 6296.996760] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6297.452723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6297.465563] alg: No test for adler32 (adler32-zlib) [ 6298.313834] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6298.404548] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6299.983178] Key type lgssc registered [ 6300.365266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6302.964090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6302.971563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.119836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6305.534319] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6308.295282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.402310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6309.595983] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6311.949730] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6313.869761] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6316.056749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6316.161888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.952295] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6320.201440] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.509981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6333.919708] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6333.921997] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.926016] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6334.944241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6334.945837] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6336.008651] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 6336.011388] Lustre: Skipped 1 previous similar message [ 6337.806323] Lustre: server umount lustre-MDT0001 complete [ 6340.064253] LustreError: 427554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.064649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.065411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.066080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6340.070823] LustreError: 427554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6345.184617] LustreError: 427553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.185104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6345.420610] Lustre: server umount lustre-OST0000 complete [ 6346.717616] Lustre: server umount lustre-MDT0000 complete [ 6351.150447] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6352.098929] Key type lgssc unregistered [ 6352.230387] LNet: 431253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6352.233400] LNetError: 431253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6352.245288] LNet: Removed LNI 192.168.206.138@tcp [ 6352.526421] Key type .llcrypt unregistered [ 6352.527338] Key type ._llcrypt unregistered [ 6360.409329] Key type ._llcrypt registered [ 6360.410253] Key type .llcrypt registered [ 6360.445345] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 6366.271426] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6366.710534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6366.722038] alg: No test for adler32 (adler32-zlib) [ 6367.577184] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6367.658905] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6369.239155] Key type lgssc registered [ 6369.615274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6380.905244] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6385.660503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6385.666167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.737174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6386.747465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6386.779659] Lustre: lustre-MDT0000: new disk, initializing [ 6386.795816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6386.800480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6388.030780] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6393.686321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.785701] Lustre: lustre-OST0000: new disk, initializing [ 6393.787838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6393.789425] Lustre: Skipped 1 previous similar message [ 6393.791349] Lustre: 436254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6393.814277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.630203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6394.633786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6394.642879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6395.810472] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6399.637871] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6405.127413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.169231] Lustre: 435239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6405.196435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6405.233316] Lustre: lustre-MDT0001: new disk, initializing [ 6405.266804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6405.278177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6405.282028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6406.462210] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6410.047966] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6410.120406] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6411.559028] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6411.632197] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6416.354202] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6416.354292] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.358124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6416.361738] Lustre: Skipped 1 previous similar message [ 6417.420365] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.38@tcp (stopping) [ 6417.422790] Lustre: Skipped 1 previous similar message [ 6419.348953] Lustre: server umount lustre-MDT0001 complete [ 6426.594046] LustreError: 435243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.598529] LustreError: 435243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6431.713175] LustreError: 435245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.719038] LustreError: 435245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6435.808076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6435.810276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6435.814640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6436.833193] LustreError: 435249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.952640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6441.953030] LustreError: 438495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6441.955996] Lustre: Skipped 1 previous similar message [ 6446.047130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6446.105304] Lustre: server umount lustre-OST0000 complete [ 6447.073208] LustreError: 435249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6453.621767] Lustre: server umount lustre-MDT0000 complete [ 6458.201474] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6459.261044] Key type lgssc unregistered [ 6459.404361] LNet: 439999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6459.407153] LNetError: 439999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6459.417329] LNet: Removed LNI 192.168.206.138@tcp [ 6459.736680] Key type .llcrypt unregistered [ 6459.738219] Key type ._llcrypt unregistered [ 6471.813848] Key type ._llcrypt registered [ 6471.815440] Key type .llcrypt registered [ 6471.855066] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 6471.954361] Key type .llcrypt unregistered [ 6471.955320] Key type ._llcrypt unregistered [ 6478.368970] Key type ._llcrypt registered [ 6478.369884] Key type .llcrypt registered [ 6478.402423] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 6478.702021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6478.741064] alg: No test for adler32 (adler32-zlib) [ 6479.585341] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 6479.665566] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 6481.239124] Key type lgssc registered [ 6481.571903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6489.715075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6489.721723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.804101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6490.816766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6490.852664] Lustre: lustre-MDT0000: new disk, initializing [ 6490.873221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.879703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6492.241862] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6496.398629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6496.444176] Lustre: 443345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6496.468153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6496.470965] Lustre: Skipped 1 previous similar message [ 6496.509712] Lustre: lustre-MDT0001: new disk, initializing [ 6496.542199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6496.554261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6496.558654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6497.932287] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6501.731367] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6503.646404] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6505.903465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.004435] Lustre: lustre-OST0000: new disk, initializing [ 6506.006363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6506.009877] Lustre: 445316:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6506.041397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.995494] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6511.851165] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6515.705062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6515.708233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6515.740309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6520.802962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.805556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.806444] Lustre: Skipped 2 previous similar messages [ 6520.810851] Lustre: Skipped 2 previous similar messages [ 6525.533419] Lustre: server umount lustre-MDT0000 complete [ 6526.833252] LustreError: 443340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316465 with bad export cookie 6656837047305830112 [ 6526.838247] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.940729] Lustre: server umount lustre-MDT0001 complete [ 6538.392086] Lustre: server umount lustre-OST0000 complete [ 6541.859906] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:14:40 (1785316480) [ 6544.342247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.481059] LustreError: 447343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.490536] LustreError: 447343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6544.501553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.832199] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6548.669983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.798094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.219146] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6552.672446] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.489295] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6569.826365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6569.913550] Lustre: Found index 0 for lustre-OST0000, updating log [ 6569.994074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.995075] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 6574.424425] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.502185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6599.648180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6599.651686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.658321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.672844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6600.674425] Lustre: Skipped 2 previous similar messages [ 6602.314915] Lustre: server umount lustre-OST0000 complete [ 6615.519823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6615.522580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.525908] Lustre: Skipped 1 previous similar message [ 6615.527476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6622.176032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6622.177918] Lustre: Skipped 3 previous similar messages [ 6629.343074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6629.392864] Lustre: server umount lustre-MDT0000 complete [ 6630.527205] LustreError: 447325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316569 with bad export cookie 6656837047305831491 [ 6630.529821] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.530680] LustreError: 447325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6630.646989] Lustre: server umount lustre-MDT0001 complete [ 6633.371150] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:16:11 (1785316571) [ 6640.415324] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [ 7213.000845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.145370] LustreError: 453770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7213.170184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.539927] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7217.420837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7217.540994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7218.936672] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7221.487524] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7223.344612] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7238.753961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.847362] Lustre: Found index 0 for lustre-OST0000, updating log [ 7238.967403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.955312] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7240.999326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7243.335395] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.025713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.029414] Lustre: Skipped 1 previous similar message [ 7245.031323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7245.034082] Lustre: Skipped 4 previous similar messages [ 7246.050042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7246.052272] Lustre: Skipped 1 previous similar message [ 7250.121970] Lustre: server umount lustre-OST0000 complete [ 7251.424221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.425267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7251.429012] Lustre: Skipped 1 previous similar message [ 7255.520342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7255.522820] Lustre: Skipped 8 previous similar messages [ 7257.496351] Lustre: server umount lustre-MDT0000 complete [ 7257.568223] LustreError: 453789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.573106] LustreError: 453789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7258.592992] LustreError: 453771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7258.598332] LustreError: 453771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7258.703163] LustreError: 453750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317197 with bad export cookie 6656837047305832562 [ 7258.706159] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7258.707180] LustreError: 453750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7258.820744] Lustre: server umount lustre-MDT0001 complete [ 7268.127189] LNet: 1 local NIs in recovery (showing 1): 192.168.206.138@tcp [ 7276.210535] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7279.583051] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7291.814155] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:27:10 (1785317230) [ 7297.780502] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7298.803697] Key type lgssc unregistered [ 7302.047342] LNet: 460425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7302.049942] LNetError: 460425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7370.661319] LNet: Removed LNI 192.168.206.138@tcp [ 7370.989529] Key type .llcrypt unregistered [ 7370.991323] Key type ._llcrypt unregistered [ 7377.716772] Key type ._llcrypt registered [ 7377.717668] Key type .llcrypt registered [ 7377.752379] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7378.060725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7378.124620] alg: No test for adler32 (adler32-zlib) [ 7378.969728] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7379.055447] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7380.631148] Key type lgssc registered [ 7380.960813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7382.280559] LNet: Added LNI 192.168.206.138@tcp100 [8/256/0/180] [ 7382.559934] LNet: Added LNI 192.168.206.138@tcp101 [8/256/0/180] [ 7382.856410] LNet: Added LNI 192.168.206.138@tcp102 [8/256/0/180] [ 7383.148710] LNet: Added LNI 192.168.206.138@tcp103 [8/256/0/180] [ 7383.438805] LNet: Added LNI 192.168.206.138@tcp104 [8/256/0/180] [ 7383.727915] LNet: Added LNI 192.168.206.138@tcp105 [8/256/0/180] [ 7383.993855] LNet: Added LNI 192.168.206.138@tcp106 [8/256/0/180] [ 7384.265915] LNet: Added LNI 192.168.206.138@tcp107 [8/256/0/180] [ 7384.540219] LNet: Added LNI 192.168.206.138@tcp108 [8/256/0/180] [ 7384.806684] LNet: Added LNI 192.168.206.138@tcp109 [8/256/0/180] [ 7385.085243] LNet: Added LNI 192.168.206.138@tcp110 [8/256/0/180] [ 7385.369797] LNet: Added LNI 192.168.206.138@tcp111 [8/256/0/180] [ 7385.651719] LNet: Added LNI 192.168.206.138@tcp112 [8/256/0/180] [ 7385.931960] LNet: Added LNI 192.168.206.138@tcp113 [8/256/0/180] [ 7386.205742] LNet: Added LNI 192.168.206.138@tcp114 [8/256/0/180] [ 7386.489814] LNet: Added LNI 192.168.206.138@tcp115 [8/256/0/180] [ 7386.773420] LNet: Added LNI 192.168.206.138@tcp116 [8/256/0/180] [ 7387.048170] LNet: Added LNI 192.168.206.138@tcp117 [8/256/0/180] [ 7387.333684] LNet: Added LNI 192.168.206.138@tcp118 [8/256/0/180] [ 7387.621671] LNet: Added LNI 192.168.206.138@tcp119 [8/256/0/180] [ 7387.910087] LNet: Added LNI 192.168.206.138@tcp120 [8/256/0/180] [ 7388.190102] LNet: Added LNI 192.168.206.138@tcp121 [8/256/0/180] [ 7388.474056] LNet: Added LNI 192.168.206.138@tcp122 [8/256/0/180] [ 7388.746465] LNet: Added LNI 192.168.206.138@tcp123 [8/256/0/180] [ 7389.033818] LNet: Added LNI 192.168.206.138@tcp124 [8/256/0/180] [ 7389.316912] LNet: Added LNI 192.168.206.138@tcp125 [8/256/0/180] [ 7389.600882] LNet: Added LNI 192.168.206.138@tcp126 [8/256/0/180] [ 7389.872365] LNet: Added LNI 192.168.206.138@tcp127 [8/256/0/180] [ 7390.142302] LNet: Added LNI 192.168.206.138@tcp128 [8/256/0/180] [ 7390.410237] LNet: Added LNI 192.168.206.138@tcp129 [8/256/0/180] [ 7390.677882] LNet: Added LNI 192.168.206.138@tcp130 [8/256/0/180] [ 7390.936704] LNet: Added LNI 192.168.206.138@tcp131 [8/256/0/180] [ 7391.200102] LNet: Added LNI 192.168.206.138@tcp132 [8/256/0/180] [ 7391.464572] LNet: Added LNI 192.168.206.138@tcp133 [8/256/0/180] [ 7391.745931] LNet: Added LNI 192.168.206.138@tcp134 [8/256/0/180] [ 7392.056896] LNet: Added LNI 192.168.206.138@tcp135 [8/256/0/180] [ 7394.351578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7394.358874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.406082] Lustre: 463423:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [ 7395.454040] LNetError: 463446:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7395.463821] LNetError: Protocol error connecting to 192.168.206.138@tcp120 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7395.466986] LNetError: Skipped 1 previous similar message [ 7395.500038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7395.511720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7395.548645] Lustre: lustre-MDT0000: new disk, initializing [ 7395.569809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.576222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7396.447275] LNet: 6 local NIs in recovery (showing 6): 192.168.206.138@tcp122, 192.168.206.138@tcp123, 192.168.206.138@tcp124, 192.168.206.138@tcp120, 192.168.206.138@tcp125, 192.168.206.138@tcp126 [ 7396.449690] LNetError: 460887:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7396.455569] LNetError: 460887:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7396.458297] LNetError: Protocol error connecting to 192.168.206.138@tcp104 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7396.461613] LNetError: Skipped 5 previous similar messages [ 7396.978356] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7398.497822] LNetError: 463455:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7398.500328] LNetError: 463455:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 7398.502596] LNetError: Protocol error connecting to 192.168.206.138@tcp125 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7398.505439] LNetError: Skipped 4 previous similar messages [ 7401.105363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.162899] Lustre: 463438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7401.192585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7401.195229] Lustre: Skipped 1 previous similar message [ 7401.234174] Lustre: lustre-MDT0001: new disk, initializing [ 7401.248379] Lustre: 464245:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [ 7401.252108] Lustre: 464245:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7401.299517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7401.310886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7401.315077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7402.592206] LNetError: 460888:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7402.595933] LNetError: Protocol error connecting to 192.168.206.138@tcp125 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7402.648726] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7405.298064] Lustre: 464737:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [ 7405.300549] Lustre: 464737:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7406.399882] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7408.296777] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7410.633439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7410.757858] Lustre: lustre-OST0000: new disk, initializing [ 7410.759449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7410.761839] Lustre: 465415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7410.784572] LNetError: 460885:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7410.788544] LNetError: Protocol error connecting to 192.168.206.138@tcp125 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7410.800552] Lustre: 465420:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [ 7410.804767] Lustre: 465420:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7410.821165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7412.836462] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7414.786544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7414.789349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7414.813949] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.138@tcp: can't add NID 192.168.206.138@tcp130: rc = -75 [ 7414.816741] Lustre: 465925:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [ 7414.842093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7416.494434] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.872559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7419.874489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.875462] LustreError: Skipped 1 previous similar message [ 7419.879568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7419.882422] Lustre: Skipped 1 previous similar message [ 7424.913935] Lustre: server umount lustre-OST0000 complete [ 7426.143903] LNetError: 463448:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7426.148090] LNetError: Protocol error connecting to 192.168.206.138@tcp125 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7427.039795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7427.043452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.046669] Lustre: Skipped 1 previous similar message [ 7427.048328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7430.112161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7430.112609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.231978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7435.233867] Lustre: Skipped 2 previous similar messages [ 7439.328632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.331685] Lustre: Skipped 1 previous similar message [ 7440.351154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7440.415771] Lustre: server umount lustre-MDT0000 complete [ 7441.636449] LustreError: 464244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317380 with bad export cookie 13901530422921799315 [ 7441.639578] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7441.641280] LustreError: 464244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7441.772705] Lustre: server umount lustre-MDT0001 complete [ 7443.552509] LNetError: 463451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.138 [ 7443.558051] LNetError: 463451:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7443.561514] LNetError: Protocol error connecting to 192.168.206.138@tcp123 at host 192.168.206.138:988: is it running a compatible version of Lustre? [ 7443.564437] LNetError: Skipped 7 previous similar messages [ 7445.448359] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7446.461735] Key type lgssc unregistered [ 7446.620847] LNet: 467327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7446.624844] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7446.627034] LNet: Removed LNI 192.168.206.138@tcp [ 7447.647124] LNet: Removed LNI 192.168.206.138@tcp100 [ 7447.648676] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7447.651036] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7447.653627] LNet: Removed LNI 192.168.206.138@tcp101 [ 7448.671149] LNet: Removed LNI 192.168.206.138@tcp102 [ 7448.672771] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7448.675111] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7448.677514] LNet: Removed LNI 192.168.206.138@tcp103 [ 7449.695166] LNet: Removed LNI 192.168.206.138@tcp104 [ 7450.719197] LNet: Removed LNI 192.168.206.138@tcp105 [ 7450.721517] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7450.725597] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7450.729532] LNet: Removed LNI 192.168.206.138@tcp106 [ 7451.743183] LNet: Removed LNI 192.168.206.138@tcp107 [ 7452.767184] LNet: Removed LNI 192.168.206.138@tcp108 [ 7453.791197] LNet: Removed LNI 192.168.206.138@tcp109 [ 7454.815169] LNet: Removed LNI 192.168.206.138@tcp110 [ 7454.817036] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7454.820714] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7454.824380] LNet: Removed LNI 192.168.206.138@tcp111 [ 7455.839186] LNet: Removed LNI 192.168.206.138@tcp112 [ 7456.863323] LNet: Removed LNI 192.168.206.138@tcp113 [ 7457.887215] LNet: Removed LNI 192.168.206.138@tcp114 [ 7458.911167] LNet: Removed LNI 192.168.206.138@tcp115 [ 7459.935144] LNet: Removed LNI 192.168.206.138@tcp116 [ 7460.959156] LNet: Removed LNI 192.168.206.138@tcp117 [ 7461.983123] LNet: Removed LNI 192.168.206.138@tcp118 [ 7463.007164] LNet: Removed LNI 192.168.206.138@tcp119 [ 7463.009300] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7463.011597] LNetError: 467327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7463.014136] LNet: Removed LNI 192.168.206.138@tcp120 [ 7464.031142] LNet: Removed LNI 192.168.206.138@tcp121 [ 7465.055161] LNet: Removed LNI 192.168.206.138@tcp122 [ 7466.079131] LNet: Removed LNI 192.168.206.138@tcp123 [ 7467.103134] LNet: Removed LNI 192.168.206.138@tcp124 [ 7467.104965] LNet: Removed LNI 192.168.206.138@tcp125 [ 7468.127185] LNet: Removed LNI 192.168.206.138@tcp126 [ 7468.128794] LNet: Removed LNI 192.168.206.138@tcp127 [ 7468.130561] LNet: Removed LNI 192.168.206.138@tcp128 [ 7468.132641] LNet: Removed LNI 192.168.206.138@tcp129 [ 7468.134146] LNet: Removed LNI 192.168.206.138@tcp130 [ 7468.135398] LNet: Removed LNI 192.168.206.138@tcp131 [ 7468.137149] LNet: Removed LNI 192.168.206.138@tcp132 [ 7468.139133] LNet: Removed LNI 192.168.206.138@tcp133 [ 7468.141478] LNet: Removed LNI 192.168.206.138@tcp134 [ 7468.149552] LNet: Removed LNI 192.168.206.138@tcp135 [ 7468.427167] Key type .llcrypt unregistered [ 7468.428393] Key type ._llcrypt unregistered [ 7476.077746] Key type ._llcrypt registered [ 7476.078833] Key type .llcrypt registered [ 7476.116479] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7482.582763] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7483.203745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7483.221070] alg: No test for adler32 (adler32-zlib) [ 7484.074749] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7484.158244] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7485.735119] Key type lgssc registered [ 7486.103303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.865728] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:30:35 (1785317435) [ 7502.498316] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7503.462432] Key type lgssc unregistered [ 7503.591217] LNet: 471385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.593127] LNetError: 471385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.599235] LNet: Removed LNI 192.168.206.138@tcp [ 7503.864274] Key type .llcrypt unregistered [ 7503.865381] Key type ._llcrypt unregistered [ 7510.813541] Key type ._llcrypt registered [ 7510.815394] Key type .llcrypt registered [ 7510.859296] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7511.448780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7511.462419] alg: No test for adler32 (adler32-zlib) [ 7512.320015] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7512.418149] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7513.999143] Key type lgssc registered [ 7514.374547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7517.635537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7517.640152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.723771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7518.734974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7518.774354] Lustre: lustre-MDT0000: new disk, initializing [ 7518.801711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.808336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7520.273886] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7524.357837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.409506] Lustre: 472578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7524.431619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7524.434169] Lustre: Skipped 1 previous similar message [ 7524.468772] Lustre: lustre-MDT0001: new disk, initializing [ 7524.500470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.513187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7524.516708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7525.812921] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7529.595540] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7531.468970] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7533.680398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.781094] Lustre: lustre-OST0000: new disk, initializing [ 7533.782861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7533.785389] Lustre: 474542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7533.813622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.864918] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7539.739906] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.042131] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing lnet_if_list [ 7542.729930] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7542.741614] LNet: Added LNI 192.168.206.138@tcp10 [8/256/0/180] [ 7542.778066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7542.780810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7542.820154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7544.541336] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7544.555298] LNet: Added LNI 192.168.206.138@tcp11 [8/256/0/180] [ 7546.022281] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7546.033459] LNet: Added LNI 192.168.206.138@tcp12 [8/256/0/180] [ 7547.542431] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7547.557921] LNet: Added LNI 192.168.206.138@tcp13 [8/256/0/180] [ 7549.025985] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7549.036959] LNet: Added LNI 192.168.206.138@tcp14 [8/256/0/180] [ 7550.574987] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7550.586162] LNet: Added LNI 192.168.206.138@tcp15 [8/256/0/180] [ 7551.932343] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7551.941752] LNet: Added LNI 192.168.206.138@tcp16 [8/256/0/180] [ 7553.279966] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7553.292857] LNet: Added LNI 192.168.206.138@tcp17 [8/256/0/180] [ 7554.698890] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7554.707418] LNet: Added LNI 192.168.206.138@tcp18 [8/256/0/180] [ 7556.282427] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7556.294660] LNet: Added LNI 192.168.206.138@tcp19 [8/256/0/180] [ 7557.915116] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7557.929684] LNet: Added LNI 192.168.206.138@tcp20 [8/256/0/180] [ 7559.613390] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7559.629634] LNet: Added LNI 192.168.206.138@tcp21 [8/256/0/180] [ 7561.342531] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7561.354368] LNet: Added LNI 192.168.206.138@tcp22 [8/256/0/180] [ 7563.188318] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7563.201735] LNet: Added LNI 192.168.206.138@tcp23 [8/256/0/180] [ 7564.809893] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7564.819717] LNet: Added LNI 192.168.206.138@tcp24 [8/256/0/180] [ 7566.525649] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7566.537826] LNet: Added LNI 192.168.206.138@tcp25 [8/256/0/180] [ 7568.401813] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7568.412894] LNet: Added LNI 192.168.206.138@tcp26 [8/256/0/180] [ 7570.310579] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7570.322264] LNet: Added LNI 192.168.206.138@tcp27 [8/256/0/180] [ 7572.307258] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7572.318238] LNet: Added LNI 192.168.206.138@tcp28 [8/256/0/180] [ 7574.295421] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7574.308486] LNet: Added LNI 192.168.206.138@tcp29 [8/256/0/180] [ 7576.323550] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7576.336897] LNet: Added LNI 192.168.206.138@tcp30 [8/256/0/180] [ 7578.486273] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7578.499626] LNet: Added LNI 192.168.206.138@tcp31 [8/256/0/180] [ 7580.732957] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7580.748025] LNet: Added LNI 192.168.206.138@tcp32 [8/256/0/180] [ 7582.897820] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7582.909284] LNet: Added LNI 192.168.206.138@tcp33 [8/256/0/180] [ 7585.136521] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7585.150929] LNet: Added LNI 192.168.206.138@tcp34 [8/256/0/180] [ 7587.470401] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7587.484878] LNet: Added LNI 192.168.206.138@tcp35 [8/256/0/180] [ 7589.816312] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7589.827601] LNet: Added LNI 192.168.206.138@tcp36 [8/256/0/180] [ 7592.114947] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7592.127899] LNet: Added LNI 192.168.206.138@tcp37 [8/256/0/180] [ 7594.674574] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7594.687740] LNet: Added LNI 192.168.206.138@tcp38 [8/256/0/180] [ 7597.093390] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7597.107235] LNet: Added LNI 192.168.206.138@tcp39 [8/256/0/180] [ 7599.696877] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7599.708526] LNet: Added LNI 192.168.206.138@tcp40 [8/256/0/180] [ 7602.104191] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7602.114366] LNet: Added LNI 192.168.206.138@tcp41 [8/256/0/180] [ 7604.228727] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7604.243336] LNet: Added LNI 192.168.206.138@tcp42 [8/256/0/180] [ 7606.823978] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7606.837749] LNet: Added LNI 192.168.206.138@tcp43 [8/256/0/180] [ 7609.417271] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7609.431924] LNet: Added LNI 192.168.206.138@tcp44 [8/256/0/180] [ 7612.007680] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7612.019489] LNet: Added LNI 192.168.206.138@tcp45 [8/256/0/180] [ 7614.381056] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7614.393854] LNet: Added LNI 192.168.206.138@tcp46 [8/256/0/180] [ 7616.790430] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7616.802923] LNet: Added LNI 192.168.206.138@tcp47 [8/256/0/180] [ 7619.124628] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7619.136745] LNet: Added LNI 192.168.206.138@tcp48 [8/256/0/180] [ 7621.663993] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7621.676324] LNet: Added LNI 192.168.206.138@tcp49 [8/256/0/180] [ 7624.163237] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7624.176631] LNet: Added LNI 192.168.206.138@tcp50 [8/256/0/180] [ 7632.352653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7632.356029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7632.360196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.913176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.914323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.916136] Lustre: Skipped 1 previous similar message [ 7634.922576] Lustre: Skipped 1 previous similar message [ 7637.669076] Lustre: server umount lustre-MDT0000 complete [ 7638.926160] LustreError: 472573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317577 with bad export cookie 3900497437851762881 [ 7638.928308] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.931874] LustreError: 472573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7639.046099] Lustre: server umount lustre-MDT0001 complete [ 7646.496744] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7649.480935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7655.327162] Lustre: 471852:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317578/real 1785317578] req@00000000a0579809 x1872041031580928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7655.334045] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7658.399107] Lustre: 471852:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317581/real 1785317581] req@00000000cf44625c x1872041031581056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317597 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7660.511169] Lustre: 471853:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317583/real 1785317583] req@00000000e5be1bc7 x1872041031581184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7663.519441] LustreError: 471850:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd3a6131 x1872041031582080/t0(0) o250->MGC192.168.206.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7663.629820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.671106] Lustre: 471854:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317587/real 1785317587] req@0000000098e1603b x1872041031581440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7665.045729] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7668.012846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.133738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7669.390096] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7670.175129] Lustre: 471851:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317592/real 1785317592] req@00000000cf3a60bc x1872041031581568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7671.630020] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7673.267512] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7673.317313] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7674.339881] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7683.552192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7683.554343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.559734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7686.609860] Lustre: server umount lustre-OST0000 complete [ 7688.671849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7688.674406] LustreError: Skipped 1 previous similar message [ 7688.675725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.679156] Lustre: Skipped 1 previous similar message [ 7688.680893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.683329] Lustre: Skipped 2 previous similar messages [ 7693.975970] Lustre: server umount lustre-MDT0000 complete [ 7694.816949] LustreError: 498730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7694.823086] LustreError: 498730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7695.252309] LustreError: 472572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317634 with bad export cookie 3900497437851764897 [ 7695.254668] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7695.257776] LustreError: 472572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7695.370462] Lustre: server umount lustre-MDT0001 complete [ 7699.129467] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7700.153466] Key type lgssc unregistered [ 7700.325361] LNet: 501502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7700.329825] LNetError: 501502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7700.334197] LNet: Removed LNI 192.168.206.138@tcp [ 7700.336348] LNet: Removed LNI 192.168.206.138@tcp10 [ 7700.338525] LNet: Removed LNI 192.168.206.138@tcp11 [ 7700.340766] LNet: Removed LNI 192.168.206.138@tcp12 [ 7700.342791] LNet: Removed LNI 192.168.206.138@tcp13 [ 7700.345062] LNet: Removed LNI 192.168.206.138@tcp14 [ 7700.347255] LNet: Removed LNI 192.168.206.138@tcp15 [ 7700.349582] LNet: Removed LNI 192.168.206.138@tcp16 [ 7700.351625] LNet: Removed LNI 192.168.206.138@tcp17 [ 7700.353917] LNet: Removed LNI 192.168.206.138@tcp18 [ 7700.355923] LNet: Removed LNI 192.168.206.138@tcp19 [ 7700.358098] LNet: Removed LNI 192.168.206.138@tcp20 [ 7700.360269] LNet: Removed LNI 192.168.206.138@tcp21 [ 7700.362531] LNet: Removed LNI 192.168.206.138@tcp22 [ 7700.364644] LNet: Removed LNI 192.168.206.138@tcp23 [ 7700.366856] LNet: Removed LNI 192.168.206.138@tcp24 [ 7700.369080] LNet: Removed LNI 192.168.206.138@tcp25 [ 7700.371169] LNet: Removed LNI 192.168.206.138@tcp26 [ 7700.373135] LNet: Removed LNI 192.168.206.138@tcp27 [ 7700.375491] LNet: Removed LNI 192.168.206.138@tcp28 [ 7700.377087] LNet: Removed LNI 192.168.206.138@tcp29 [ 7700.378475] LNet: Removed LNI 192.168.206.138@tcp30 [ 7700.380155] LNet: Removed LNI 192.168.206.138@tcp31 [ 7700.381996] LNet: Removed LNI 192.168.206.138@tcp32 [ 7700.384275] LNet: Removed LNI 192.168.206.138@tcp33 [ 7700.386652] LNet: Removed LNI 192.168.206.138@tcp34 [ 7700.388223] LNet: Removed LNI 192.168.206.138@tcp35 [ 7700.389613] LNet: Removed LNI 192.168.206.138@tcp36 [ 7700.391092] LNet: Removed LNI 192.168.206.138@tcp37 [ 7700.392316] LNet: Removed LNI 192.168.206.138@tcp38 [ 7700.394166] LNet: Removed LNI 192.168.206.138@tcp39 [ 7700.395360] LNet: Removed LNI 192.168.206.138@tcp40 [ 7700.396478] LNet: Removed LNI 192.168.206.138@tcp41 [ 7700.397561] LNet: Removed LNI 192.168.206.138@tcp42 [ 7700.398838] LNet: Removed LNI 192.168.206.138@tcp43 [ 7700.400814] LNet: Removed LNI 192.168.206.138@tcp44 [ 7700.402430] LNet: Removed LNI 192.168.206.138@tcp45 [ 7700.404254] LNet: Removed LNI 192.168.206.138@tcp46 [ 7700.406578] LNet: Removed LNI 192.168.206.138@tcp47 [ 7700.410589] LNet: Removed LNI 192.168.206.138@tcp48 [ 7700.412541] LNet: Removed LNI 192.168.206.138@tcp49 [ 7700.420277] LNet: Removed LNI 192.168.206.138@tcp50 [ 7700.729406] Key type .llcrypt unregistered [ 7700.731483] Key type ._llcrypt unregistered [ 7707.572210] Key type ._llcrypt registered [ 7707.573595] Key type .llcrypt registered [ 7707.612959] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7713.927469] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7714.261500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7714.318404] alg: No test for adler32 (adler32-zlib) [ 7715.178437] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7715.273711] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7716.855096] Key type lgssc registered [ 7717.224087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7728.847552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7728.853590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.938487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7729.950524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7729.994628] Lustre: lustre-MDT0000: new disk, initializing [ 7730.019971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7730.028142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7731.446961] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7735.646335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.692402] Lustre: 504673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7735.719958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7735.722781] Lustre: Skipped 1 previous similar message [ 7735.760557] Lustre: lustre-MDT0001: new disk, initializing [ 7735.793928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7735.805427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7735.811130] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7737.181733] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7740.824187] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7742.709028] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7745.241588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.322294] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7745.353950] Lustre: lustre-OST0000: new disk, initializing [ 7745.356325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7745.358940] Lustre: 506687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7745.392348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7747.301932] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7751.052426] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7753.415158] Lustre: server umount lustre-OST0000 complete [ 7755.744969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7755.752217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7755.755988] Lustre: Skipped 1 previous similar message [ 7760.853214] Lustre: server umount lustre-MDT0000 complete [ 7762.061135] LustreError: 505467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317700 with bad export cookie 11272170522365862166 [ 7762.064270] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.065524] LustreError: 505467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7762.177656] Lustre: server umount lustre-MDT0001 complete [ 7766.013869] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7767.025614] Key type lgssc unregistered [ 7767.149408] LNet: 508583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7767.152184] LNetError: 508583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7767.159381] LNet: Removed LNI 192.168.206.138@tcp [ 7767.461862] Key type .llcrypt unregistered [ 7767.463114] Key type ._llcrypt unregistered [ 7775.374879] Key type ._llcrypt registered [ 7775.375800] Key type .llcrypt registered [ 7775.417530] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7775.786594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7775.859032] alg: No test for adler32 (adler32-zlib) [ 7776.713119] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7776.806484] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7778.391156] Key type lgssc registered [ 7778.765764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.348823] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7781.355408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7782.492851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7783.830482] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7786.640026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.759091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7788.187185] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7790.726388] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7792.654395] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.985403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.097727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.134911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7796.138593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7797.125537] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7799.655969] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7807.415228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7813.817086] loop0: detected capacity change from 0 to 204800000 [ 7813.878847] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7813.883542] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7813.889785] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7815.889255] loop0: detected capacity change from 0 to 204800000 [ 7815.968250] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7815.974555] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7815.981822] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7817.684761] loop0: detected capacity change from 0 to 204800000 [ 7819.686772] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7819.735898] Lustre: fs15246-OST0000: new disk, initializing [ 7819.738500] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7819.742604] Lustre: 513447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7819.763696] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.912123] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7825.937335] loop1: detected capacity change from 0 to 204800000 [ 7827.967673] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.009887] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7828.033889] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7828.073245] Lustre: fs15246-MDT0000: new disk, initializing [ 7828.127934] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.139323] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7828.188178] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7829.216766] Lustre: 514567:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000184ba2ec x1872041308358656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785317784 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7829.583924] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7838.303266] LustreError: 514567:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7838.308643] LustreError: 514567:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7838.313445] LustreError: 514567:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7841.251373] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.256345] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7843.602498] Lustre: server umount fs15246-MDT0000 complete [ 7846.372799] LustreError: 509823:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.377249] LustreError: 509823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7846.788988] Lustre: server umount fs15246-OST0000 complete [ 7851.490646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7851.495061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7851.498690] Lustre: Skipped 1 previous similar message [ 7856.467080] Lustre: server umount lustre-OST0000 complete [ 7858.656124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7858.659890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.664494] Lustre: Skipped 1 previous similar message [ 7858.667301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.728394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7861.729163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.955695] Lustre: server umount lustre-MDT0000 complete [ 7865.300325] LustreError: 509810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317804 with bad export cookie 14401341293559827672 [ 7865.304670] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7865.305410] LustreError: 509810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7865.439596] Lustre: server umount lustre-MDT0001 complete [ 7869.317619] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7870.473615] Key type lgssc unregistered [ 7870.621568] LNet: 517659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7870.623981] LNetError: 517659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7870.634430] LNet: Removed LNI 192.168.206.138@tcp [ 7871.005736] Key type .llcrypt unregistered [ 7871.007065] Key type ._llcrypt unregistered [ 7878.812662] Key type ._llcrypt registered [ 7878.813801] Key type .llcrypt registered [ 7878.858533] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7879.288125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7879.347626] alg: No test for adler32 (adler32-zlib) [ 7880.237119] Lustre: Lustre: Build Version: 2.17.55_22_g38c21b0 [ 7880.344227] LNet: Added LNI 192.168.206.138@tcp [8/256/0/180] [ 7881.935189] Key type lgssc registered [ 7882.427636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7895.290748] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7898.368140] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7910.766314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7910.772152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.857935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7911.870687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7911.909834] Lustre: lustre-MDT0000: new disk, initializing [ 7911.933144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.939660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7913.475859] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7917.845062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7917.891391] Lustre: 522093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7917.916048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7917.919032] Lustre: Skipped 1 previous similar message [ 7917.955330] Lustre: lustre-MDT0001: new disk, initializing [ 7917.995645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7918.009418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7918.013550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7919.534672] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7923.396472] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7925.384792] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7927.497910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7927.597476] Lustre: lustre-OST0000: new disk, initializing [ 7927.599300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7927.601764] Lustre: 524062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7927.627454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.402781] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_default_debug -1 all [ 7933.012058] Lustre: DEBUG MARKER: oleg638-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7934.390962] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7935.480844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7935.483985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7935.516612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7936.507168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7937.972209] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7938.054670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7940.578035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7940.578751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.580431] LustreError: Skipped 1 previous similar message [ 7940.581106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7940.584330] Lustre: Skipped 1 previous similar message [ 7944.786469] Lustre: server umount lustre-OST0000 complete [ 7948.768018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7948.771317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.776710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.778817] Lustre: Skipped 1 previous similar message [ 7950.816188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7950.816744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.215243] Lustre: server umount lustre-MDT0000 complete [ 7953.453209] LustreError: 522889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317892 with bad export cookie 638097664166228641 [ 7953.456540] LustreError: MGC192.168.206.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7953.457488] LustreError: 522889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7953.562653] Lustre: server umount lustre-MDT0001 complete [ 7961.257853] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing set_hostid [ 7964.452460] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing load_modules_local [ 7974.612920] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7736 sec ========== 05:38:32 (1785317912) [ 7975.139295] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:38:33 (1785317913) === [ 7976.280946] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:38:34 (1785317914) === [ 7989.810294] Lustre: DEBUG MARKER: oleg638-server.virtnet: executing unload_modules_local [ 7990.852957] Key type lgssc unregistered [ 7990.979410] LNet: 530705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7990.982089] LNetError: 530705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7990.991331] LNet: Removed LNI 192.168.206.138@tcp [ 7991.323403] Key type .llcrypt unregistered [ 7991.325127] Key type ._llcrypt unregistered