[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486928360 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003308] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022382] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026820] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031110] *** VALIDATE tmpfs *** [ 0.033128] *** VALIDATE proc *** [ 0.034253] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.036282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043671] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.045842] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052026] ... fixed-purpose events: 3 [ 0.053017] ... event mask: 000000070000000f [ 0.054304] rcu: Hierarchical SRCU implementation. [ 0.056502] smp: Bringing up secondary CPUs ... [ 0.057620] x86: Booting SMP configuration: [ 0.058033] .... node #0, CPUs: #1 #2 #3 [ 0.061500] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.191485] node 0 deferred pages initialised in 126ms [ 0.195371] devtmpfs: initialized [ 0.196238] x86/mm: Memory block size: 128MB [ 0.198835] gcov: version magic: 0x41383552 [ 0.200294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.201096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.202277] pinctrl core: initialized pinctrl subsystem [ 0.203224] [ 0.203849] ************************************************************* [ 0.204034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205024] ** ** [ 0.206021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207015] ** ** [ 0.208015] ** This means that this kernel is built to expose internal ** [ 0.209013] ** IOMMU data structures, which may compromise security on ** [ 0.210016] ** your system. ** [ 0.211017] ** ** [ 0.212015] ** If you see this message and you are not debugging the ** [ 0.213012] ** kernel, report this immediately to your vendor! ** [ 0.214020] ** ** [ 0.215014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.216014] ************************************************************* [ 0.217793] NET: Registered protocol family 16 [ 0.218485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.219071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222010] cpuidle: using governor menu [ 0.224286] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227576] PCI: Using configuration type 1 for base access [ 0.229153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239214] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.240046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.241390] cryptd: max_cpu_qlen set to 1000 [ 0.243506] ACPI: Added _OSI(Module Device) [ 0.244033] ACPI: Added _OSI(Processor Device) [ 0.245025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.251804] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258382] ACPI: Interpreter enabled [ 0.260088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262019] ACPI: Using IOAPIC for interrupt routing [ 0.264187] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.267656] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.278000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288157] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293515] acpiphp: Slot [2] registered [ 0.295218] acpiphp: Slot [5] registered [ 0.296203] acpiphp: Slot [6] registered [ 0.298199] acpiphp: Slot [7] registered [ 0.300186] acpiphp: Slot [8] registered [ 0.301139] acpiphp: Slot [9] registered [ 0.303249] acpiphp: Slot [10] registered [ 0.305189] acpiphp: Slot [3] registered [ 0.308190] acpiphp: Slot [4] registered [ 0.309271] acpiphp: Slot [11] registered [ 0.311220] acpiphp: Slot [12] registered [ 0.313112] acpiphp: Slot [13] registered [ 0.314115] acpiphp: Slot [14] registered [ 0.316200] acpiphp: Slot [15] registered [ 0.317089] acpiphp: Slot [16] registered [ 0.318115] acpiphp: Slot [17] registered [ 0.320115] acpiphp: Slot [18] registered [ 0.322104] acpiphp: Slot [19] registered [ 0.323118] acpiphp: Slot [20] registered [ 0.325117] acpiphp: Slot [21] registered [ 0.326119] acpiphp: Slot [22] registered [ 0.328109] acpiphp: Slot [23] registered [ 0.330124] acpiphp: Slot [24] registered [ 0.331208] acpiphp: Slot [25] registered [ 0.333134] acpiphp: Slot [26] registered [ 0.335131] acpiphp: Slot [27] registered [ 0.336081] acpiphp: Slot [28] registered [ 0.338128] acpiphp: Slot [29] registered [ 0.339090] acpiphp: Slot [30] registered [ 0.340142] acpiphp: Slot [31] registered [ 0.342126] PCI host bridge to bus 0000:00 [ 0.343019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348048] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354112] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360301] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.364176] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.367491] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378744] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.383526] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.387033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.390033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.396521] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398894] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401157] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404898] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.410015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.423020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.428015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436581] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466039] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.473025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.497018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.507662] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.521017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.529025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.569448] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.576018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.583020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.597023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.605190] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.620024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.636024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.656026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.680017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.693026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.711019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.721844] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.732025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.739021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.760025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.770000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.773423] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.775469] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778447] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.781231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.786137] iommu: Default domain type: Passthrough [ 0.788568] SCSI subsystem initialized [ 0.791237] ACPI: bus type USB registered [ 0.792127] usbcore: registered new interface driver usbfs [ 0.794201] usbcore: registered new interface driver hub [ 0.796145] usbcore: registered new device driver usb [ 0.798223] pps_core: LinuxPPS API ver. 1 registered [ 0.800017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.803065] PTP clock support registered [ 0.805308] EDAC MC: Ver: 3.0.0 [ 0.807630] PCI: Using ACPI for IRQ routing [ 0.810009] NetLabel: Initializing [ 0.812059] NetLabel: domain hash size = 128 [ 0.813000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.815124] NetLabel: unlabeled traffic allowed by default [ 0.817261] vgaarb: loaded [ 0.819259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.821019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.828000] clocksource: Switched to clocksource kvm-clock [ 0.959062] VFS: Disk quotas dquot_6.6.0 [ 0.960777] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.964295] *** VALIDATE ramfs *** [ 0.965930] *** VALIDATE hugetlbfs *** [ 0.967601] pnp: PnP ACPI init [ 0.970457] pnp: PnP ACPI: found 6 devices [ 0.990111] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.993640] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.996282] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.998848] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.001595] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.004045] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.007043] NET: Registered protocol family 2 [ 1.009784] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016229] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021643] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.030334] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.034072] TCP: Hash tables configured (established 65536 bind 65536) [ 1.038824] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.042032] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045311] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.048263] NET: Registered protocol family 1 [ 1.052092] RPC: Registered named UNIX socket transport module. [ 1.054880] RPC: Registered udp transport module. [ 1.057321] RPC: Registered tcp transport module. [ 1.059785] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062883] NET: Registered protocol family 44 [ 1.064655] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.066794] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.068917] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.071149] PCI: CLS 0 bytes, default 64 [ 1.073043] Unpacking initramfs... [ 3.197378] debug: unmapping init [mem 0xffff9cadbcc54000-0xffff9cadbffbffff] [ 3.209324] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.213117] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.220251] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.504935] Initialise system trusted keyrings [ 5.506787] Key type blacklist registered [ 5.515059] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.546089] zbud: loaded [ 5.569584] *** VALIDATE nfs *** [ 5.570989] *** VALIDATE nfs4 *** [ 5.575102] pstore: using deflate compression [ 5.586858] Platform Keyring initialized [ 6.041539] NET: Registered protocol family 38 [ 6.043121] Key type asymmetric registered [ 6.045995] Asymmetric key parser 'x509' registered [ 6.051677] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.056622] io scheduler mq-deadline registered [ 6.061174] io scheduler kyber registered [ 6.065161] io scheduler bfq registered [ 6.068826] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.074244] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.081704] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.086547] ACPI: Power Button [PWRF] [ 6.102626] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.114618] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.144953] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.157685] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.185134] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.224158] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.265748] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.273580] Non-volatile memory driver v1.3 [ 6.276519] Linux agpgart interface v0.103 [ 6.398955] virtio_blk virtio1: [vda] 145168 512-byte logical blocks (74.3 MB/70.9 MiB) [ 6.412865] vda: detected capacity change from 0 to 74326016 [ 6.472014] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.478970] vdb: detected capacity change from 0 to 1073741824 [ 6.530349] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.537548] vdc: detected capacity change from 0 to 2621440000 [ 6.568435] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.572854] vdd: detected capacity change from 0 to 2621440000 [ 6.625111] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.639410] vde: detected capacity change from 0 to 4294967296 [ 6.668831] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.673691] vdf: detected capacity change from 0 to 4294967296 [ 6.693494] libphy: Fixed MDIO Bus: probed [ 6.709371] usbcore: registered new interface driver usbserial_generic [ 6.716266] usbserial: USB Serial support registered for generic [ 6.720379] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.728619] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.732538] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.738539] mousedev: PS/2 mouse device common for all mice [ 6.750259] rtc_cmos 00:05: RTC can wake from S4 [ 6.769939] rtc_cmos 00:05: registered as rtc0 [ 6.785289] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.786595] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.796759] intel_pstate: CPU model not supported [ 6.820584] hid: raw HID events driver (C) Jiri Kosina [ 6.820783] usbcore: registered new interface driver usbhid [ 6.820787] usbhid: USB HID core driver [ 6.820936] drop_monitor: Initializing network drop monitor service [ 6.821109] Initializing XFRM netlink socket [ 6.821452] NET: Registered protocol family 10 [ 6.839733] Segment Routing with IPv6 [ 6.848806] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.856651] NET: Registered protocol family 17 [ 6.857722] mpls_gso: MPLS GSO support [ 6.873660] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.896951] RAS: Correctable Errors collector initialized. [ 6.899845] AVX version of gcm_enc/dec engaged. [ 6.903089] AES CTR mode by8 optimization enabled [ 7.118701] sched_clock: Marking stable (7118590413, 0)->(8125732844, -1007142431) [ 7.153131] registered taskstats version 1 [ 7.173986] Loading compiled-in X.509 certificates [ 7.178099] zswap: loaded using pool lzo/zbud [ 7.254351] Key type big_key registered [ 7.288242] Key type encrypted registered [ 7.295601] ima: No TPM chip found, activating TPM-bypass! [ 7.301449] ima: Allocated hash algorithm: sha1 [ 7.303399] ima: No architecture policies found [ 7.309596] evm: Initialising EVM extended attributes: [ 7.317235] evm: security.selinux [ 7.320449] evm: security.ima [ 7.323728] evm: security.capability [ 7.327957] evm: HMAC attrs: 0x1 [ 7.341634] rtc_cmos 00:05: setting system clock to 2026-07-04 03:59:05 UTC (1783137545) [ 7.372395] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 7.406988] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 7.427245] Write protecting the kernel read-only data: 28672k [ 7.441607] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 7.449213] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 7.614469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.639539] systemd[1]: Detected virtualization kvm. [ 7.642237] systemd[1]: Detected architecture x86-64. [ 7.653576] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.764192] systemd[1]: No hostname configured. [ 7.771923] systemd[1]: Set hostname to . [ 7.779169] random: systemd: uninitialized urandom read (16 bytes read) [ 7.783988] systemd[1]: Initializing machine ID from random generator. [ 8.190675] random: systemd: uninitialized urandom read (16 bytes read) [ 8.205274] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.231714] random: systemd: uninitialized urandom read (16 bytes read) [ 8.245780] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 8.274835] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.185277] device-mapper: uevent: version 1.0.3 [ 10.198699] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 11.920646] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.026945] random: fast init done Starting dracut initqueue hook... [ 12.105416] scsi host0: ata_piix [ 12.134900] scsi host1: ata_piix [ 12.137967] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.145359] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.849685] random: crng init done [ 16.851238] random: 7 urandom warning(s) missed due to ratelimiting [ 17.912972] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.071592] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.335254] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.126338] SELinux: Disabled at runtime. [ 24.292052] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.310661] systemd[1]: Detected virtualization kvm. [ 24.313287] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.886660] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.892809] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.907838] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.915859] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.922730] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.936922] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.959555] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ 26.300785] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.713794] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.552112] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.591692] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.128349] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.179770] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.838796] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 36.605746] NFS: Registering the id_resolver key type [ 36.609930] Key type id_resolver registered [ 36.613655] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 54.693013] hrtimer: interrupt took 4279666 ns [ 111.634468] libcfs: loading out-of-tree module taints kernel. [ 111.749539] Key type ._llcrypt registered [ 111.755133] Key type .llcrypt registered [ 111.887706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 132.896568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 135.180941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 135.201210] alg: No test for adler32 (adler32-zlib) [ 136.838691] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 138.080741] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 139.944207] Key type lgssc registered [ 141.976466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 163.433258] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 201.903585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 214.803258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.838258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.185948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.216131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.356273] Lustre: lustre-MDT0000: new disk, initializing [ 216.457609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.482506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.575781] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.552470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.685342] Lustre: 6534:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 235.708114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 235.711925] Lustre: Skipped 1 previous similar message [ 235.791258] Lustre: lustre-MDT0001: new disk, initializing [ 235.875477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.921402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 235.937540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 240.399667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.526571] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.222146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.540234] Lustre: lustre-OST0000: new disk, initializing [ 256.544747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.550292] Lustre: 8471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.615712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.667953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.679283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.748807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.139287] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.965734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.073511] Lustre: lustre-OST0001: new disk, initializing [ 278.082930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.089553] Lustre: 9545:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.192051] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 287.209473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.774336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 287.799362] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 287.873840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.558404] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 313.061381] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 320.309795] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 325.617296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 331.244371] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 334.325565] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 337.113962] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 338.885129] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 4 00:04:36 EDT 2026 [ 361.503735] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 363.232580] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 379.872913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.880609] Lustre: Skipped 1 previous similar message [ 379.890083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.900625] Lustre: Skipped 1 previous similar message [ 384.494242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 384.506481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.518871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.048194] Lustre: server umount lustre-MDT0000 complete [ 390.031078] LustreError: 9544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137928 with bad export cookie 14932990453714282280 [ 390.033082] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.040175] LustreError: 9544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 390.115876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 390.129678] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 390.133518] Lustre: Skipped 4 previous similar messages [ 395.233263] LustreError: 9014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.234168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.254523] LustreError: 9014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 395.281760] Lustre: Skipped 1 previous similar message [ 396.464513] Lustre: server umount lustre-MDT0001 complete [ 400.472504] Lustre: server umount lustre-OST0000 complete [ 404.484717] Lustre: server umount lustre-OST0001 complete [ 412.461623] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 423.562936] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 468.019778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.414418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 468.464827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 468.553059] Lustre: lustre-MDT0000: new disk, initializing [ 468.661580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.700663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 473.657575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 486.058341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.290409] Lustre: 15102:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 486.413360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 486.419087] Lustre: Skipped 1 previous similar message [ 486.563319] Lustre: lustre-MDT0001: new disk, initializing [ 486.746051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.840450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 486.859245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 493.038995] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 502.865351] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 510.179798] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 518.837187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.512725] Lustre: lustre-OST0000: new disk, initializing [ 519.516788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 519.520892] Lustre: 17073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.626840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.032883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 521.039840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 521.231163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 528.983975] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 540.300354] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 545.409802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 545.735216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 550.934412] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 551.193250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 557.026186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.027080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.027088] Lustre: Skipped 1 previous similar message [ 557.030680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.058305] LustreError: Skipped 1 previous similar message [ 557.148576] Lustre: Skipped 1 previous similar message [ 559.857341] Lustre: server umount lustre-OST0000 complete [ 567.274221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.305602] Lustre: Skipped 1 previous similar message [ 567.312323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 567.322203] Lustre: Skipped 1 previous similar message [ 573.092738] Lustre: server umount lustre-MDT0000 complete [ 577.523957] LustreError: 15110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.551606] LustreError: 15110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 578.773659] LustreError: 15096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138116 with bad export cookie 14932990453714283813 [ 578.800635] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 579.339808] Lustre: server umount lustre-MDT0001 complete [ 584.257179] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:08:41 (1783138121) [ 586.673720] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 588.829472] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:08:46 (1783138126) [ 595.748518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.107221] LustreError: 19484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.231083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.957685] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 601.573980] LustreError: 19485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.691486] LustreError: 19484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.958517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.062300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 622.756301] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 628.406079] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 635.450654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.817274] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 635.830736] Lustre: Skipped 2 previous similar messages [ 643.556372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 652.230322] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 661.486205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 661.499093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.344846] Lustre: server umount lustre-OST0000 complete [ 671.717523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 671.737672] LustreError: Skipped 1 previous similar message [ 671.746102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 671.753314] Lustre: Skipped 3 previous similar messages [ 677.834595] Lustre: server umount lustre-MDT0000 complete [ 677.870200] LustreError: 19485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.824130] LustreError: 19466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138219 with bad export cookie 14932990453714284695 [ 681.829231] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.837856] LustreError: 19466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 682.387146] Lustre: server umount lustre-MDT0001 complete [ 695.664622] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 699.849945] Key type lgssc unregistered [ 700.241992] LNet: 23015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.267206] LNetError: 23015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.285995] LNet: Removed LNI 192.168.204.146@tcp [ 702.903542] Key type .llcrypt unregistered [ 702.905371] Key type ._llcrypt unregistered [ 731.435252] Key type ._llcrypt registered [ 731.440182] Key type .llcrypt registered [ 731.585545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 733.057360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.264173] alg: No test for adler32 (adler32-zlib) [ 734.527644] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 734.848135] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 736.600236] Key type lgssc registered [ 738.043550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 748.018944] loop: module loaded [ 748.115242] loop0: detected capacity change from 0 to 204800000 [ 748.414529] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 748.431951] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 749.013498] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.379430] loop0: detected capacity change from 0 to 204800000 [ 756.674341] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.699146] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 756.723721] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 762.518324] loop0: detected capacity change from 0 to 204800000 [ 769.243965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 769.275599] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.610579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 770.682877] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 770.770536] Lustre: lustre-MDT0000: new disk, initializing [ 770.870987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.906737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 775.614667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 787.044223] loop1: detected capacity change from 0 to 204800000 [ 793.993541] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 794.566578] Lustre: lustre-OST07c6: new disk, initializing [ 794.581412] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 794.590328] Lustre: Skipped 1 previous similar message [ 794.601569] Lustre: 26207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 794.737333] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 803.010365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 803.018855] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 803.073300] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 804.347323] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 810.669355] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 810.685458] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 810.704426] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 810.716733] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 810.725667] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 810.734332] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 830.551535] Lustre: 25194:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 830.568080] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 830.579851] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 830.592550] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 830.608958] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 830.620068] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 835.963868] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 835.996114] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 836.027234] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 836.035303] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.038744] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 836.042171] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.046484] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 836.049919] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.054417] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 836.057890] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 836.062114] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 836.066392] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 841.996342] Lustre: 26800:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 842.009090] Lustre: 26800:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 842.018786] Lustre: 26800:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 842.047246] Lustre: 26800:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 842.056512] Lustre: 26800:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 842.078038] Lustre: 26800:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 842.089318] Lustre: 26800:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 842.102146] Lustre: 26800:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 842.119764] Lustre: 26800:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 842.129638] Lustre: 26800:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 842.138234] Lustre: 26800:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 842.150629] Lustre: 26800:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 850.962396] Lustre: 26800:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 850.977117] Lustre: 26800:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 850.986169] Lustre: 26800:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 850.996602] Lustre: 26800:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.004841] Lustre: 26800:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 851.013307] Lustre: 26800:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.024228] Lustre: 26800:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 851.029908] Lustre: 26800:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.038532] Lustre: 26800:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 851.047323] Lustre: 26800:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 851.056896] Lustre: 26800:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 851.065827] Lustre: 26800:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 859.184470] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 859.200561] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 859.218075] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 859.234328] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 859.250953] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 859.261367] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 859.273800] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 859.283793] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 859.301505] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 859.311751] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 859.318766] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 859.329665] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 910.312139] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.339952] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 915.426557] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 916.064661] Lustre: server umount lustre-OST07c6 complete [ 928.886355] Lustre: server umount lustre-MDT0000 complete [ 949.386612] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 952.981675] Key type lgssc unregistered [ 953.348738] LNet: 28771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.362278] LNetError: 28771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.404176] LNet: Removed LNI 192.168.204.146@tcp [ 954.359965] Key type .llcrypt unregistered [ 954.362359] Key type ._llcrypt unregistered [ 980.414307] Key type ._llcrypt registered [ 980.417166] Key type .llcrypt registered [ 980.558764] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 981.872391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.066504] alg: No test for adler32 (adler32-zlib) [ 983.248328] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 983.497624] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 985.216184] Key type lgssc registered [ 986.686710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 995.035038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 995.064054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.474965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.583862] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1006.053222] LustreError: 30047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.067945] LustreError: 30047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1011.171980] LustreError: 30046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.121299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.596304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.718272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1026.266138] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1032.712680] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.796969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1042.376757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.906290] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1059.315982] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1065.930348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1081.339227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.340399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.363062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.372901] Lustre: Skipped 1 previous similar message [ 1083.135346] Lustre: server umount lustre-OST0000 complete [ 1089.505657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.519574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.532766] Lustre: Skipped 1 previous similar message [ 1089.540913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.557270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.559335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.744533] Lustre: server umount lustre-MDT0000 complete [ 1096.683229] LustreError: 31153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.709266] LustreError: 31153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1098.923784] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138637 with bad export cookie 5794784354649661753 [ 1098.932173] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.937430] LustreError: 30026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1099.352803] Lustre: server umount lustre-MDT0001 complete [ 1112.798482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1116.405955] Key type lgssc unregistered [ 1116.783772] LNet: 33785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.803725] LNetError: 33785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.849236] LNet: Removed LNI 192.168.204.146@tcp [ 1118.159255] Key type .llcrypt unregistered [ 1118.161899] Key type ._llcrypt unregistered [ 1139.509927] Key type ._llcrypt registered [ 1139.513937] Key type .llcrypt registered [ 1139.647217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1141.185743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.280308] alg: No test for adler32 (adler32-zlib) [ 1142.420823] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1142.694948] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1144.449242] Key type lgssc registered [ 1145.854975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.415168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.472307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.174505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.375116] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1167.336400] LustreError: 35069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.370031] LustreError: 35069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1172.377832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1172.449161] LustreError: 35070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1172.821092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1177.140284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1184.837174] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1191.103760] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1198.581993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1199.119959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.229144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1206.536869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1215.747816] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1227.745951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1227.772926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.792053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.383254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1233.395701] Lustre: Skipped 3 previous similar messages [ 1233.543078] Lustre: server umount lustre-OST0000 complete [ 1238.505565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.536528] Lustre: Skipped 1 previous similar message [ 1238.546436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1238.550421] Lustre: Skipped 1 previous similar message [ 1243.511816] Lustre: server umount lustre-MDT0000 complete [ 1243.635137] LustreError: 36504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.655805] LustreError: 36504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1248.039431] LustreError: 35052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138786 with bad export cookie 12466773237820668527 [ 1248.044595] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.054152] LustreError: 35052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.591669] Lustre: server umount lustre-MDT0001 complete [ 1260.017248] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1263.083711] Key type lgssc unregistered [ 1263.368273] LNet: 38611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.373362] LNetError: 38611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.401249] LNet: Removed LNI 192.168.204.146@tcp [ 1264.403859] Key type .llcrypt unregistered [ 1264.406743] Key type ._llcrypt unregistered [ 1289.593788] Key type ._llcrypt registered [ 1289.595827] Key type .llcrypt registered [ 1289.712380] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1291.033268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.168084] alg: No test for adler32 (adler32-zlib) [ 1292.330282] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1292.564083] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1294.232180] Key type lgssc registered [ 1295.556177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.118272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.165449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.853672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.844424] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1315.301845] LustreError: 40029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.320112] LustreError: 40029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1320.247145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.428399] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1320.776712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.153339] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1334.094326] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1340.740214] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1347.821986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.547401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.622604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1356.906408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1367.088747] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1374.323238] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1375.209069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.234024] Lustre: Skipped 2 previous similar messages [ 1375.235459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.261951] Lustre: Skipped 2 previous similar messages [ 1379.426897] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1379.685205] Lustre: server umount lustre-MDT0000 complete [ 1380.325571] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.350672] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1384.209577] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138922 with bad export cookie 15415402501899160885 [ 1384.212231] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.230421] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1384.536162] LustreError: 40026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1384.567566] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 1385.442323] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.478855] LustreError: 40025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.479331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1389.513702] LustreError: 40025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1389.520949] Lustre: Skipped 1 previous similar message [ 1390.805928] Lustre: server umount lustre-MDT0001 complete [ 1412.453734] Lustre: server umount lustre-OST0000 complete [ 1427.656429] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1430.380954] Key type lgssc unregistered [ 1430.651498] LNet: 43888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.666358] LNetError: 43888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.725465] LNet: Removed LNI 192.168.204.146@tcp [ 1432.676350] Key type .llcrypt unregistered [ 1432.678444] Key type ._llcrypt unregistered [ 1456.752137] Key type ._llcrypt registered [ 1456.754551] Key type .llcrypt registered [ 1456.861918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1457.727697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1457.736552] alg: No test for adler32 (adler32-zlib) [ 1458.840715] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1459.150239] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1460.880229] Key type lgssc registered [ 1462.123477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1469.917286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1469.943966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1471.527700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.485099] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1481.717123] LustreError: 45308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1481.740901] LustreError: 45308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1484.532078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.952606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1489.076986] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1497.194458] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1503.347196] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1511.032868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1511.460132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1516.607285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1517.563929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1524.599868] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1534.946965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1534.964678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.968836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1534.986773] Lustre: Skipped 1 previous similar message [ 1534.998077] Lustre: Skipped 1 previous similar message [ 1536.635935] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1540.143891] Lustre: server umount lustre-OST0000 complete [ 1560.549680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.577150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1563.762439] Lustre: server umount lustre-MDT0000 complete [ 1565.671739] LustreError: 45307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1565.691786] LustreError: 45307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1567.425155] LustreError: 45289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139105 with bad export cookie 103822351460152957 [ 1567.430144] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1567.438498] LustreError: 45289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1574.010891] Lustre: server umount lustre-MDT0001 complete [ 1586.758133] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1590.276683] Key type lgssc unregistered [ 1590.556635] LNet: 49016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.576965] LNetError: 49016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.598069] LNet: Removed LNI 192.168.204.146@tcp [ 1591.702169] Key type .llcrypt unregistered [ 1591.708801] Key type ._llcrypt unregistered [ 1619.031151] Key type ._llcrypt registered [ 1619.036461] Key type .llcrypt registered [ 1619.209859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1620.790323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1620.843395] alg: No test for adler32 (adler32-zlib) [ 1621.875680] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1622.072992] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1623.720273] Key type lgssc registered [ 1625.699680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.164587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.198952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.792100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.178913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1647.074597] LustreError: 50433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.097358] LustreError: 50433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1652.195311] LustreError: 50434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.441468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.882592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1657.678452] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1665.904218] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.011781] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1678.873895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1679.261591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1682.367232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1686.188756] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1694.661510] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1704.396746] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1707.695950] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1719.523520] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-04 0h27m36s [ 1720.863496] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1722.980222] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1723.364159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.390174] Lustre: Skipped 2 previous similar messages [ 1724.055681] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 1724.066576] Lustre: Skipped 3 previous similar messages [ 1728.998737] Lustre: server umount lustre-MDT0000 complete [ 1729.116046] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.721552] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1735.249669] LustreError: 50429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1735.258653] LustreError: 50429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1737.126119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.244030] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1737.481186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.545417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1738.538923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1742.047076] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1745.372127] Lustre: DEBUG MARKER: Wait for df (36574) ... [ 1746.685224] Lustre: DEBUG MARKER: done [ 1753.059725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.074505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.104673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.110394] Lustre: Skipped 3 previous similar messages [ 1754.087910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.093677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.650407] Lustre: server umount lustre-OST0000 complete [ 1764.325728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.329680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1764.339708] Lustre: Skipped 1 previous similar message [ 1764.340924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.384517] Lustre: Skipped 3 previous similar messages [ 1766.822706] Lustre: server umount lustre-MDT0000 complete [ 1771.314859] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139309 with bad export cookie 6910878639905887635 [ 1771.349525] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.819383] Lustre: server umount lustre-MDT0001 complete [ 1784.614799] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1787.070035] Key type lgssc unregistered [ 1787.412858] LNet: 55361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.426681] LNetError: 55361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.449482] LNet: Removed LNI 192.168.204.146@tcp [ 1788.516233] Key type .llcrypt unregistered [ 1788.521390] Key type ._llcrypt unregistered [ 1829.465291] Key type ._llcrypt registered [ 1829.467622] Key type .llcrypt registered [ 1829.604596] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1830.780040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.815716] alg: No test for adler32 (adler32-zlib) [ 1831.839222] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 1832.092847] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1833.776192] Key type lgssc registered [ 1834.678315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1841.963207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1841.991115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.275265] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1843.299636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1843.450610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.779560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1857.014118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.118428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.156122] Lustre: 57568:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1857.454215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.768833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1868.878748] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1874.275686] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1880.584519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1880.764147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1880.952650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.454158] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1889.472621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1894.955421] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1902.362850] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-04 0h30m39s [ 1904.614974] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1913.751667] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1916.449214] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1916.591436] LustreError: 60590:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1937.040837] Lustre: lustre-MDT0000: Client 4ff7fbde-eded-46a3-8964-2e4e02748222 (at 192.168.204.46@tcp) reconnecting [ 1956.625233] LustreError: 60590:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1958.651370] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1960.892942] Lustre: DEBUG MARKER: done [ 1966.563890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.591148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.816785] Lustre: server umount lustre-OST0000 complete [ 1975.273312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1975.285260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.316114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.420817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.437479] Lustre: Skipped 2 previous similar messages [ 1979.300282] Lustre: server umount lustre-MDT0000 complete [ 1983.186138] LustreError: 57561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139521 with bad export cookie 10374453046615356358 [ 1983.186413] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.194981] LustreError: 57561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1993.866092] Lustre: server umount lustre-MDT0001 complete [ 2004.817864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2007.692885] Key type lgssc unregistered [ 2008.068350] LNet: 61904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.086415] LNetError: 61904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.105304] LNet: Removed LNI 192.168.204.146@tcp [ 2009.059229] Key type .llcrypt unregistered [ 2009.063745] Key type ._llcrypt unregistered [ 2052.713911] Key type ._llcrypt registered [ 2052.718712] Key type .llcrypt registered [ 2052.855309] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2053.833351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.844871] alg: No test for adler32 (adler32-zlib) [ 2054.916860] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 2055.109433] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2056.784228] Key type lgssc registered [ 2057.817659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2060.857168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2060.877160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.191711] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2062.236145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2062.378843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.248943] Lustre: Failing over lustre-MDT0000 [ 2063.517144] Lustre: server umount lustre-MDT0000 complete [ 2070.193100] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:33:27 (1783139607) [ 2076.895459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.164238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.782997] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2091.576725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.719974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2091.765539] Lustre: 64707:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2092.036403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.290477] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2105.792754] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.919764] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2118.289731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2118.571266] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2118.834437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.967153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2125.564610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2132.808671] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2138.837305] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2148.325349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.342504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.356021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.860429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2149.866445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.096120] Lustre: server umount lustre-MDT0000 complete [ 2157.004338] LustreError: 64698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139695 with bad export cookie 9176839250653925185 [ 2157.019493] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2157.390286] Lustre: server umount lustre-MDT0001 complete [ 2161.127819] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2165.196537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2173.154715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.416641] Lustre: 63628:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139698/real 1783139698] req@00000000ffb26233 x1869757381911552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2176.458349] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.469211] Lustre: Skipped 1 previous similar message [ 2178.528238] Lustre: 63626:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139700/real 1783139700] req@0000000055d406b1 x1869757381911680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2181.664746] Lustre: 63626:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139703/real 1783139703] req@00000000e7949669 x1869757381911808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.625476] LustreError: 63624:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000055d406b1 x1869757381912576/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2182.953336] LustreError: 66554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.986580] LustreError: 66554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2183.076252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.182783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2184.100418] Lustre: 63628:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139705/real 1783139705] req@00000000cd442a31 x1869757381912064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.058661] LustreError: 66555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2185.076142] LustreError: 66555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2186.088152] LustreError: 66554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.105690] LustreError: 66554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2187.913582] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2188.128208] Lustre: 63627:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139710/real 1783139710] req@00000000dff54463 x1869757381912192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139726 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2191.332719] LustreError: 68476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.354301] LustreError: 68476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2196.373569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.453389] LustreError: 66555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.475219] LustreError: 66555:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2196.699098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2201.105903] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2202.097697] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2205.167277] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2207.261947] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2212.292771] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2219.168773] Lustre: server umount lustre-MDT0000 complete [ 2220.523514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.533932] LustreError: 68477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.534156] Lustre: Skipped 2 previous similar messages [ 2220.560593] LustreError: 68477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2222.316471] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139760 with bad export cookie 9176839250653927460 [ 2222.328357] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.331065] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2224.588829] Lustre: server umount lustre-MDT0001 complete [ 2230.936765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2239.986278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.928168] Lustre: 63625:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139763/real 1783139763] req@00000000cfbeded6 x1869757381937664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2240.967864] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.137151] LustreError: 63624:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000092682c99 x1869757381938816/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.416899] LustreError: 66554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2247.481617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.537394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2251.460345] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2258.470500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.843622] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2262.411615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2269.319550] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.198654] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2284.539000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.549342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.566264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.574086] Lustre: Skipped 2 previous similar messages [ 2286.566382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.577873] Lustre: Skipped 2 previous similar messages [ 2288.597845] Lustre: server umount lustre-MDT0000 complete [ 2291.683960] LustreError: 71154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.719967] LustreError: 71154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2292.877196] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139831 with bad export cookie 9176839250653928839 [ 2292.881799] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2292.884681] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2293.481627] Lustre: server umount lustre-MDT0001 complete [ 2300.240125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2303.674844] Lustre: DEBUG MARKER: files compared the same [ 2315.272805] Lustre: server umount lustre-OST0000 complete [ 2336.789710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2340.140275] Key type lgssc unregistered [ 2340.487618] LNet: 74430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.500409] LNetError: 74430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2341.542834] LNet: Removed LNI 192.168.204.146@tcp [ 2342.664131] Key type .llcrypt unregistered [ 2342.667333] Key type ._llcrypt unregistered [ 2367.292136] Key type ._llcrypt registered [ 2367.294361] Key type .llcrypt registered [ 2367.401554] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2368.512639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.617263] alg: No test for adler32 (adler32-zlib) [ 2369.708912] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 2369.975723] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2371.680203] Key type lgssc registered [ 2372.999321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.020740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.052306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.740346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2387.711614] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2392.034676] LustreError: 75849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.055357] LustreError: 75849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2395.655272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.917088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2400.302241] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2407.046743] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2412.296538] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2418.472595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.748637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.140510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug +malloc all [ 2426.934668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2432.461346] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2442.212667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.217068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2442.252594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.274643] LustreError: Skipped 1 previous similar message [ 2446.818630] Lustre: server umount lustre-OST0000 complete [ 2452.449452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.455942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.464256] Lustre: Skipped 2 previous similar messages [ 2452.472034] Lustre: Skipped 1 previous similar message [ 2456.813486] Lustre: server umount lustre-MDT0000 complete [ 2457.568813] LustreError: 75845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.598932] LustreError: 75845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2460.428473] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139998 with bad export cookie 8328098114884912159 [ 2460.440394] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.440897] LustreError: 75831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2460.714931] Lustre: server umount lustre-MDT0001 complete [ 2470.928691] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2473.625443] Key type lgssc unregistered [ 2473.858686] LNet: 79383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.865523] LNetError: 79383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.876954] LNet: Removed LNI 192.168.204.146@tcp [ 2474.728087] Key type .llcrypt unregistered [ 2474.730950] Key type ._llcrypt unregistered [ 2499.988706] Key type ._llcrypt registered [ 2499.991651] Key type .llcrypt registered [ 2500.090388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2501.168343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2501.196653] alg: No test for adler32 (adler32-zlib) [ 2502.345295] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 2502.820330] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2504.576924] Key type lgssc registered [ 2506.307214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2515.756306] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2515.813143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2531.680571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2536.800361] LustreError: 80774:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 2563.552228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.324683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2578.413239] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2587.122226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.237872] Lustre: *** cfs_fail_loc=706, val=0*** [ 2587.631200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.716840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2592.666744] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2592.746067] LustreError: 80799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2592.760438] LustreError: 80799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2597.859129] LustreError: 80800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.899741] LustreError: 82137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.920776] LustreError: 82137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2601.687185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.033261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2606.699341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2615.916635] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.730387] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2627.560797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.562019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.563974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.590369] Lustre: Skipped 1 previous similar message [ 2629.859273] Lustre: server umount lustre-OST0000 complete [ 2637.797143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.798239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.812786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.832562] Lustre: Skipped 2 previous similar messages [ 2639.640699] Lustre: server umount lustre-MDT0000 complete [ 2642.862609] LustreError: 80780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140181 with bad export cookie 17469546955211032581 [ 2642.878780] LustreError: 80780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2642.880424] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.176192] Lustre: server umount lustre-MDT0001 complete [ 2652.711219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2655.472499] Key type lgssc unregistered [ 2655.794511] LNet: 84389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.804574] LNetError: 84389:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.872664] LNet: Removed LNI 192.168.204.146@tcp [ 2657.789393] Key type .llcrypt unregistered [ 2657.791202] Key type ._llcrypt unregistered [ 2684.322978] Key type ._llcrypt registered [ 2684.324493] Key type .llcrypt registered [ 2684.504334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2686.035186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2686.094612] alg: No test for adler32 (adler32-zlib) [ 2687.553968] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 2687.933126] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2689.672531] Key type lgssc registered [ 2691.671816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2703.213732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2703.267329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.246788] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2719.664050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.243527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.679929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2725.345716] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.369796] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2730.478808] LustreError: 86498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.498432] LustreError: 86498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2733.713466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.996623] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.026190] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2734.056910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.185163] LustreError: 86502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.217492] LustreError: 86502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2740.823713] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2744.291178] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.304739] LustreError: 86499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2749.937986] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.951021] Lustre: MGS: Client cbda101f-294c-4cc0-a752-c272abeebfbe (at 0@lo) reconnecting [ 2749.964947] LustreError: 86513:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2750.207776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2750.301781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2756.117516] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2770.920667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.929778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.940499] LustreError: Skipped 1 previous similar message [ 2770.950853] Lustre: Skipped 1 previous similar message [ 2770.951682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.039350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.052024] Lustre: Skipped 2 previous similar messages [ 2776.649428] Lustre: server umount lustre-OST0000 complete [ 2786.279718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2786.283089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.306318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.319705] Lustre: Skipped 1 previous similar message [ 2787.584730] Lustre: server umount lustre-MDT0000 complete [ 2791.312540] Lustre: server umount lustre-MDT0001 complete [ 2794.837139] Lustre: server umount MGS complete [ 2804.359447] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:45:42 (1783140342) [ 2831.807444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 2841.004134] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2873.767821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.724716] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2888.484800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.696659] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2888.744972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2888.837256] Lustre: lustre-MDT0001: new disk, initializing [ 2888.942266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.964493] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2888.974202] Lustre: 93669:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2888.988561] LustreError: 93669:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2893.827392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2906.287454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2906.670307] Lustre: lustre-OST0000: new disk, initializing [ 2906.678114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2906.692507] Lustre: 94436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2906.791946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.492607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2913.826181] Lustre: 94810:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008831be35 x1869758045659648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783140467 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2921.450654] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2928.396292] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2928.414756] Lustre: MGS: Client 79d29aa2-42a6-4f41-87e2-c7e6da77fd49 (at 0@lo) reconnecting [ 2928.517951] Lustre: 92938:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.535439] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2928.558165] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2928.605393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2928.749509] Lustre: lustre-MDT0000: new disk, initializing [ 2928.917695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2928.941040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2928.953637] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2930.019144] Lustre: 95506:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000524cbeb8 x1869758045666432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783140484 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2934.605557] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2935.203418] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2939.245071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2939.256304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2939.342764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2963.427891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.448601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2963.468466] Lustre: Skipped 1 previous similar message [ 2963.476090] Lustre: Skipped 1 previous similar message [ 2968.547018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2968.557255] Lustre: Skipped 1 previous similar message [ 2976.227682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2976.402968] Lustre: server umount lustre-OST0000 complete [ 2983.906108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.907166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2983.910524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2983.910536] Lustre: Skipped 3 previous similar messages [ 2983.923919] Lustre: Skipped 1 previous similar message [ 2986.588985] Lustre: server umount lustre-MDT0000 complete [ 2989.031277] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.057453] LustreError: 93645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2996.821357] Lustre: server umount lustre-MDT0001 complete [ 3000.949442] Lustre: server umount MGS complete [ 3012.769973] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:49:10 (1783140550) [ 3030.523674] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3033.319501] Key type lgssc unregistered [ 3033.601635] LNet: 98005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3033.609222] LNetError: 98005:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3033.637880] LNet: Removed LNI 192.168.204.146@tcp [ 3034.430514] Key type .llcrypt unregistered [ 3034.437339] Key type ._llcrypt unregistered [ 3049.759505] Key type ._llcrypt registered [ 3049.761761] Key type .llcrypt registered [ 3049.858902] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3051.061912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3051.108436] alg: No test for adler32 (adler32-zlib) [ 3052.163295] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3052.440295] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3054.168230] Key type lgssc registered [ 3055.183707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3062.448015] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3062.480539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.959017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.130283] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3074.533950] LustreError: 99219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.546334] LustreError: 99219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3075.251414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.595239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.267386] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3085.566983] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3091.130067] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3097.446358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.847683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.016325] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3109.110302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3110.279174] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3120.609470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.613102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.639542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3124.706730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3124.713353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3124.736826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.849259] Lustre: server umount lustre-OST0000 complete [ 3131.875094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3131.888310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.909428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3131.921858] Lustre: Skipped 1 previous similar message [ 3134.952413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.960483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.985400] Lustre: Skipped 1 previous similar message [ 3140.065540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.073851] Lustre: Skipped 1 previous similar message [ 3144.160300] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3144.338700] Lustre: server umount lustre-MDT0000 complete [ 3145.185817] LustreError: 101663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.204698] LustreError: 101663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3147.447300] LustreError: 99198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140685 with bad export cookie 16590117199649198457 [ 3147.452942] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.462790] LustreError: 99198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3147.784311] Lustre: server umount lustre-MDT0001 complete [ 3156.413321] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3158.898537] Key type lgssc unregistered [ 3159.184869] LNet: 102753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.195054] LNetError: 102753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3159.215793] LNet: Removed LNI 192.168.204.146@tcp [ 3160.027136] Key type .llcrypt unregistered [ 3160.031709] Key type ._llcrypt unregistered [ 3174.788656] Key type ._llcrypt registered [ 3174.791767] Key type .llcrypt registered [ 3174.915330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3175.930705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.973693] alg: No test for adler32 (adler32-zlib) [ 3177.085413] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3177.301045] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3178.984250] Key type lgssc registered [ 3180.169396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.423684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.433531] LustreError: 103835:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3186.435711] Lustre: Skipped 1 previous similar message [ 3186.451943] LustreError: 103832:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3186.457305] LustreError: 103835:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 3186.469680] LustreError: 103835:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3186.478422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.479805] LustreError: 103835:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3187.954919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.195744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.436653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3199.746871] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3205.229398] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lsmod [ 3207.977847] LustreError: 105031:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3208.046732] LustreError: 105042:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3208.051738] LustreError: 105031:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4935 [ 3208.058363] LustreError: 105031:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3208.066733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.069737] LustreError: 105031:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3208.344927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.795074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.800037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.805075] Lustre: Skipped 2 previous similar messages [ 3213.820530] Lustre: Skipped 2 previous similar messages [ 3217.095177] Lustre: server umount lustre-MDT0000 complete [ 3218.915893] LustreError: 103874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.937114] LustreError: 103874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3219.940419] LustreError: 103874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.967152] LustreError: 103874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3221.003153] LustreError: 103851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140759 with bad export cookie 13155422486447363839 [ 3221.017107] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3221.022288] LustreError: 103851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3222.112306] LustreError: 104305:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000082cf32c4 x1869758558391552/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3222.216978] Lustre: server umount lustre-MDT0001 complete [ 3235.721638] Lustre: server umount lustre-OST0000 complete [ 3243.250907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.725042] LustreError: 106104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.785658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.814322] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3249.124176] LustreError: 106105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.227486] LustreError: 106104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.604729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.903983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.746330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3264.833709] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3269.437505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3275.416781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3275.635279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.585095] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3287.047795] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3287.928756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3298.276565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.291451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3298.296896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3298.297958] Lustre: Skipped 1 previous similar message [ 3298.325369] Lustre: Skipped 1 previous similar message [ 3303.394683] LustreError: 107926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.412697] LustreError: 107926:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3303.482535] Lustre: server umount lustre-OST0000 complete [ 3308.515632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.555082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.575887] Lustre: Skipped 1 previous similar message [ 3312.934935] Lustre: server umount lustre-MDT0000 complete [ 3316.186834] LustreError: 106087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140854 with bad export cookie 13155422486447364525 [ 3316.200130] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.201347] LustreError: 106087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3316.533803] Lustre: server umount lustre-MDT0001 complete [ 3324.936295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3327.343673] Key type lgssc unregistered [ 3327.587944] LNet: 109640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.595937] LNetError: 109640:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3327.611697] LNet: Removed LNI 192.168.204.146@tcp [ 3328.501646] Key type .llcrypt unregistered [ 3328.506142] Key type ._llcrypt unregistered [ 3348.534894] Key type ._llcrypt registered [ 3348.536840] Key type .llcrypt registered [ 3348.630915] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3349.551885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3349.606056] alg: No test for adler32 (adler32-zlib) [ 3350.692605] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3350.916123] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3352.568211] Key type lgssc registered [ 3353.559539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3360.170488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3360.198396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.642603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.811277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3370.984039] LustreError: 111045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.004902] LustreError: 111045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3373.334501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.642149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3377.705337] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3383.761752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3388.685285] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3394.935213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.283384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.256672] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3407.604872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3408.069544] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3413.307301] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3416.472763] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3423.203548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.209264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.229049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.239047] Lustre: Skipped 1 previous similar message [ 3426.241837] Lustre: server umount lustre-OST0000 complete [ 3429.858588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3429.865523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.879234] Lustre: Skipped 1 previous similar message [ 3429.883618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.441856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.443141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.554627] Lustre: server umount lustre-MDT0000 complete [ 3438.565942] LustreError: 111046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.600663] LustreError: 111046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3438.943423] LustreError: 113575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140977 with bad export cookie 13524197634086157614 [ 3438.946565] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3438.955496] LustreError: 113575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3439.307407] Lustre: server umount lustre-MDT0001 complete [ 3447.564235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3449.690881] Key type lgssc unregistered [ 3449.904873] LNet: 114678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3449.912250] LNetError: 114678:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3449.923736] LNet: Removed LNI 192.168.204.146@tcp [ 3450.549815] Key type .llcrypt unregistered [ 3450.554735] Key type ._llcrypt unregistered [ 3464.177131] Key type ._llcrypt registered [ 3464.179221] Key type .llcrypt registered [ 3464.273220] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3465.017154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3465.086143] alg: No test for adler32 (adler32-zlib) [ 3466.108228] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3466.309823] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3467.968179] Key type lgssc registered [ 3468.852516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3475.177494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3475.190351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.500121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.269282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3486.694982] LustreError: 115879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.716316] LustreError: 115879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3487.235101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.626293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3491.511204] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3497.638621] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3502.030347] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3508.054363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3508.398283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.900253] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3517.879462] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3521.715833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3532.263156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.264293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3532.266324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3532.283281] Lustre: Skipped 1 previous similar message [ 3534.470599] Lustre: server umount lustre-OST0000 complete [ 3537.379196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.385739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.400948] Lustre: Skipped 1 previous similar message [ 3537.401914] Lustre: Skipped 1 previous similar message [ 3542.497610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3542.508234] Lustre: Skipped 2 previous similar messages [ 3543.343061] Lustre: server umount lustre-MDT0000 complete [ 3545.845205] LustreError: 117721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141084 with bad export cookie 11293135307125553298 [ 3545.853176] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3545.859450] LustreError: 117721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3546.097629] Lustre: server umount lustre-MDT0001 complete [ 3554.202900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3556.248573] Key type lgssc unregistered [ 3556.470731] LNet: 119409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3556.480310] LNetError: 119409:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3556.498053] LNet: Removed LNI 192.168.204.146@tcp [ 3557.131765] Key type .llcrypt unregistered [ 3557.134096] Key type ._llcrypt unregistered [ 3575.027965] Key type ._llcrypt registered [ 3575.032020] Key type .llcrypt registered [ 3575.120996] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3575.920114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3575.962405] alg: No test for adler32 (adler32-zlib) [ 3577.038861] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3577.264222] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3578.920165] Key type lgssc registered [ 3579.883968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3587.106109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3587.129601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.498628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.736697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3598.836822] LustreError: 120803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.858524] LustreError: 120803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3601.941185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.358216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3606.647525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3613.288981] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.459376] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3624.200647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.525425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.135994] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3633.772688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3636.860126] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3643.375565] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3646.219221] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3646.226877] Lustre: Skipped 2 previous similar messages [ 3649.752730] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3649.765934] Lustre: Skipped 2 previous similar messages [ 3651.137198] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3651.156547] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3655.716242] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3655.724885] Lustre: Skipped 2 previous similar messages [ 3657.114226] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.46@tcp 192.168.0.[2,10]@tcp [ 3662.958683] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.46/24@tcp in log lustre-MDT0000 [ 3662.964139] Lustre: Skipped 2 previous similar messages [ 3666.709370] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 3666.712976] Lustre: Skipped 1 previous similar message [ 3669.990257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.992464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3669.993209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3670.006436] Lustre: Skipped 1 previous similar message [ 3670.022984] Lustre: Skipped 1 previous similar message [ 3675.112756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3675.121863] Lustre: Skipped 1 previous similar message [ 3675.877251] Lustre: server umount lustre-OST0000 complete [ 3679.205511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3679.211145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3679.234382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.238747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3685.348024] LustreError: 120799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.361956] LustreError: 120799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3685.394457] Lustre: server umount lustre-MDT0000 complete [ 3688.562970] LustreError: 123435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141226 with bad export cookie 2842682715046615475 [ 3688.570977] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3688.573690] LustreError: 123435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3688.967322] Lustre: server umount lustre-MDT0001 complete [ 3697.019275] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3699.077972] Key type lgssc unregistered [ 3699.290386] LNet: 125545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3699.297237] LNetError: 125545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3699.315467] LNet: Removed LNI 192.168.204.146@tcp [ 3700.018900] Key type .llcrypt unregistered [ 3700.020868] Key type ._llcrypt unregistered [ 3719.850761] Key type ._llcrypt registered [ 3719.853159] Key type .llcrypt registered [ 3719.931950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3720.530020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3720.544366] alg: No test for adler32 (adler32-zlib) [ 3721.531480] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3721.707360] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3723.336227] Key type lgssc registered [ 3724.220979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3729.923684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3729.941907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.235555] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3731.264814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.272335] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.46/24@tcp [ 3734.198493] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3740.475351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.699529] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3740.721262] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.46/24@tcp [ 3740.767143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3743.685452] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3748.657353] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3752.457966] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3757.000309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3757.219317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.135762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3766.324736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3776.696135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3786.211816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.219991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.238488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.237210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.238417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3787.249316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.043671] Lustre: server umount lustre-OST0000 complete [ 3796.966463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3796.975698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.988367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3796.993407] Lustre: Skipped 1 previous similar message [ 3798.883943] Lustre: server umount lustre-MDT0000 complete [ 3801.672199] LustreError: 127074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141339 with bad export cookie 9329701833642338034 [ 3801.672647] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.679214] LustreError: 127074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3801.911451] Lustre: server umount lustre-MDT0001 complete [ 3810.326690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 3812.643412] Key type lgssc unregistered [ 3812.924560] LNet: 130704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.935875] LNetError: 130704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3812.953473] LNet: Removed LNI 192.168.204.146@tcp [ 3813.629583] Key type .llcrypt unregistered [ 3813.631846] Key type ._llcrypt unregistered [ 3833.283530] Key type ._llcrypt registered [ 3833.285947] Key type .llcrypt registered [ 3833.388289] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3844.570551] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3845.394994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3845.430385] alg: No test for adler32 (adler32-zlib) [ 3846.434186] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 3846.598767] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 3848.244128] Key type lgssc registered [ 3849.115863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3875.709469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3875.729116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.927253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3876.952125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3877.021556] Lustre: lustre-MDT0000: new disk, initializing [ 3877.085512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.103330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3880.887650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3889.523888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.617141] Lustre: 134083:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3889.675602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3889.678668] Lustre: Skipped 1 previous similar message [ 3889.744241] Lustre: lustre-MDT0001: new disk, initializing [ 3889.822492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.850664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3889.859772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3892.955821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3899.879204] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3903.877699] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3909.115351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3909.319326] Lustre: lustre-OST0000: new disk, initializing [ 3909.323332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3909.328311] Lustre: 136051:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3909.374024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.655941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3915.310478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3915.322107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3915.395893] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3920.798685] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3925.613481] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3925.885466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3933.279909] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3933.501766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3936.237789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.254524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.273063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.279589] Lustre: Skipped 1 previous similar message [ 3940.850295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.852722] Lustre: Skipped 1 previous similar message [ 3945.960607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.963862] Lustre: Skipped 1 previous similar message [ 3950.049249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3950.293245] Lustre: server umount lustre-OST0000 complete [ 3961.339917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.371622] Lustre: Skipped 1 previous similar message [ 3961.388130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3965.778539] Lustre: server umount lustre-MDT0000 complete [ 3971.554027] LustreError: 134094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3971.586052] LustreError: 134094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3972.289650] LustreError: 134877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141510 with bad export cookie 14877084634349912508 [ 3972.309991] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3972.311072] LustreError: 134877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3973.094091] Lustre: server umount lustre-MDT0001 complete [ 3987.685939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.282133] LustreError: 138271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.404416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.571763] LustreError: 138272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.854353] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3998.720165] LustreError: 138271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.813935] LustreError: 138272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.103954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.560760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4010.759660] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4019.453676] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4026.040270] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4034.665048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4035.261723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.004813] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4052.881801] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4058.171534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4059.497391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4064.468918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4064.791694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4073.813983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4074.099523] Lustre: lustre-OST0001: new disk, initializing [ 4074.107406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4074.127561] Lustre: 141649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4074.239276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4079.788954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4079.824053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4080.018940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4081.866683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4091.308299] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4091.571966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4097.117257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4097.393551] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4104.600394] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894784abe800.ost_server_uuid 50 [ 4106.644604] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff894784abe800.ost_server_uuid in IDLE state after 0 sec [ 4115.939865] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.948127] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.966387] Lustre: Skipped 1 previous similar message [ 4115.975953] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4115.984318] LustreError: Skipped 1 previous similar message [ 4115.987884] Lustre: Skipped 2 previous similar messages [ 4117.963992] Lustre: server umount lustre-OST0001 complete [ 4121.069749] LustreError: 140096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4121.086418] LustreError: 140096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4122.594787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4122.606196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.620793] Lustre: Skipped 1 previous similar message [ 4125.666041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4125.677127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.688265] Lustre: Skipped 1 previous similar message [ 4128.545791] Lustre: server umount lustre-OST0000 complete [ 4133.857078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.877863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.909283] Lustre: Skipped 1 previous similar message [ 4139.092944] Lustre: server umount lustre-MDT0000 complete [ 4141.539140] LustreError: 142815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.568085] LustreError: 142815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4142.903079] LustreError: 141648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141681 with bad export cookie 14877084634349913390 [ 4142.907915] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.920331] LustreError: 141648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4143.288502] Lustre: server umount lustre-MDT0001 complete [ 4164.255544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4167.450811] Key type lgssc unregistered [ 4167.763882] LNet: 144653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4167.774979] LNetError: 144653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4167.793975] LNet: Removed LNI 192.168.204.146@tcp [ 4168.742090] Key type .llcrypt unregistered [ 4168.749209] Key type ._llcrypt unregistered [ 4213.259277] Key type ._llcrypt registered [ 4213.264852] Key type .llcrypt registered [ 4213.387608] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4229.896639] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4231.836533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4231.882041] alg: No test for adler32 (adler32-zlib) [ 4233.117662] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 4233.438144] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4235.202029] Key type lgssc registered [ 4236.657794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4275.686722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4275.717888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.050399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4277.089614] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4277.147493] Lustre: lustre-MDT0000: new disk, initializing [ 4277.214103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.226972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4281.651581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4294.626443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.816767] Lustre: 148812:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4294.908863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4294.913427] Lustre: Skipped 1 previous similar message [ 4295.056573] Lustre: lustre-MDT0001: new disk, initializing [ 4295.212642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.255833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4295.266490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4300.479121] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4312.103524] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4318.351621] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4327.658308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.089099] Lustre: lustre-OST0000: new disk, initializing [ 4328.093415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4328.101545] Lustre: 150784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4328.240508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4336.285444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4337.777033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4337.787724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4337.964542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4347.093171] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4352.875727] Lustre: Failing over lustre-OST0000 [ 4353.505790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.507771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.531772] LustreError: Skipped 1 previous similar message [ 4353.535858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.564610] Lustre: Skipped 1 previous similar message [ 4353.595860] Lustre: Skipped 1 previous similar message [ 4355.103703] Lustre: server umount lustre-OST0000 complete [ 4375.318614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4375.841846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4375.869207] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4381.775936] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4383.145844] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4387.381488] Lustre: Failing over lustre-MDT0000 [ 4387.829052] Lustre: server umount lustre-MDT0000 complete [ 4388.322822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.346518] Lustre: Skipped 2 previous similar messages [ 4393.448398] LustreError: 150147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.474730] LustreError: 150147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4394.468206] LustreError: 148818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.503286] LustreError: 148818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4397.137543] LustreError: 148819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.587861] LustreError: 149853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.615585] LustreError: 149853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4404.711853] LustreError: 148822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.737934] LustreError: 148822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4405.728173] Lustre: 147032:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141926/real 1783141926] req@00000000c3940dee x1869759665715200/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1783141942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4405.756258] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4408.480320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.530823] LustreError: 148819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.572280] LustreError: 148819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4415.217744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4415.314266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4415.391074] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4418.665128] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4420.597330] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4420.630546] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4420.730869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4421.501072] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4435.941834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.950680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4435.975540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4435.976773] LustreError: Skipped 1 previous similar message [ 4436.001948] Lustre: Skipped 1 previous similar message [ 4437.366606] Lustre: server umount lustre-OST0000 complete [ 4446.177603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4446.179824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.187773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.228903] Lustre: Skipped 2 previous similar messages [ 4447.785499] Lustre: server umount lustre-MDT0000 complete [ 4451.298615] LustreError: 148823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.317469] LustreError: 148823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4452.183691] LustreError: 148803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141990 with bad export cookie 5264057059287459303 [ 4452.186993] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4452.195834] LustreError: 148803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4452.741717] Lustre: server umount lustre-MDT0001 complete [ 4465.270568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 4468.226884] Key type lgssc unregistered [ 4468.520179] LNet: 154439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4468.530282] LNetError: 154439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4469.561306] LNet: Removed LNI 192.168.204.146@tcp [ 4470.461708] Key type .llcrypt unregistered [ 4470.468776] Key type ._llcrypt unregistered [ 4510.181574] Key type ._llcrypt registered [ 4510.187522] Key type .llcrypt registered [ 4510.362743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4511.815949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4511.855559] alg: No test for adler32 (adler32-zlib) [ 4512.996525] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 4513.279907] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 4514.952960] Key type lgssc registered [ 4516.189397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4525.540165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4525.577295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.069076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.126670] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4537.331961] LustreError: 155965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.351870] LustreError: 155965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4542.297929] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4542.440915] LustreError: 155966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.497273] LustreError: 155994:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4546.513988] LustreError: 155994:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4546.521237] LustreError: 155994:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4547.088946] Lustre: server umount lustre-MDT0000 complete [ 4562.940539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4573.318467] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4610.857090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4611.057657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4611.094778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4611.179611] Lustre: lustre-MDT0000: new disk, initializing [ 4611.236424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.249861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4616.269793] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4627.683950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.828083] Lustre: 159297:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4627.895777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4627.903191] Lustre: Skipped 1 previous similar message [ 4627.997026] Lustre: lustre-MDT0001: new disk, initializing [ 4628.112613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4628.162923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4628.172211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4633.777689] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4644.643969] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.627319] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4657.904232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.260702] Lustre: lustre-OST0000: new disk, initializing [ 4658.265683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4658.271635] Lustre: 161268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4658.407299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.431491] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4665.925936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4665.938320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4666.077958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4675.279836] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.711526] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4681.541152] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:16:59 (1783142219) [ 4922.338474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4922.344692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4922.348934] Lustre: Skipped 2 previous similar messages [ 4922.361681] Lustre: Skipped 2 previous similar messages [ 4927.456795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.464368] Lustre: Skipped 2 previous similar messages [ 4928.356690] Lustre: server umount lustre-MDT0000 complete [ 4932.284470] LustreError: 160102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142470 with bad export cookie 6767159370124588514 [ 4932.288255] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4932.294466] LustreError: 160102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4932.577599] LustreError: 161537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.592927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.611060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4937.700213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4937.700213] LustreError: 161537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.781160] Lustre: server umount lustre-MDT0001 complete [ 4948.930934] Lustre: server umount lustre-OST0000 complete [ 4957.769178] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4966.642794] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4999.771790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.160060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5000.241654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5000.352365] Lustre: lustre-MDT0000: new disk, initializing [ 5000.423056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.437981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5004.519582] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5014.981482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.111835] Lustre: 165928:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5015.179659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5015.183280] Lustre: Skipped 1 previous similar message [ 5015.249884] Lustre: lustre-MDT0001: new disk, initializing [ 5015.360452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5015.402757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5015.416385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5019.368904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5028.382193] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5033.567763] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5039.610226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5039.882938] Lustre: lustre-OST0000: new disk, initializing [ 5039.887902] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5039.894102] Lustre: 167896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5039.979693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.362380] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5045.831090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5045.851086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5045.963749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5053.101454] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5057.396547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5057.582891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5061.885748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5062.164931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5066.724530] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5066.726451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.729750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5066.771144] Lustre: Skipped 1 previous similar message [ 5070.214137] Lustre: server umount lustre-OST0000 complete [ 5076.967147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5076.974883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.977665] LustreError: Skipped 1 previous similar message [ 5076.980371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5076.980378] Lustre: Skipped 1 previous similar message [ 5077.026687] Lustre: Skipped 1 previous similar message [ 5079.786403] Lustre: server umount lustre-MDT0000 complete [ 5082.083272] LustreError: 165936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.098723] LustreError: 165936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5082.902335] LustreError: 165920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142621 with bad export cookie 6767159370124747337 [ 5082.912814] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5082.915344] LustreError: 165920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5083.268633] Lustre: server umount lustre-MDT0001 complete [ 5091.891168] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:23:49 (1783142629) [ 5106.775376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5115.283624] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5145.643938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5145.840431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5145.859138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5145.927854] Lustre: lustre-MDT0000: new disk, initializing [ 5146.013472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.030281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5149.476631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5158.174448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5158.253695] Lustre: 172652:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5158.453928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5161.805350] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5169.528729] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5174.125539] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5179.793905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.001609] Lustre: lustre-OST0000: new disk, initializing [ 5180.005137] Lustre: Skipped 1 previous similar message [ 5180.009424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5180.013595] Lustre: Skipped 2 previous similar messages [ 5180.017804] Lustre: 174625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5185.487739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5186.639307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5186.648484] Lustre: Skipped 1 previous similar message [ 5186.663503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5186.722413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5193.888266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.409197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.413297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5202.427928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5202.438759] Lustre: Skipped 2 previous similar messages [ 5214.180136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5214.302626] Lustre: server umount lustre-OST0000 complete [ 5217.765488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.780706] Lustre: Skipped 2 previous similar messages [ 5223.452451] Lustre: server umount lustre-MDT0000 complete [ 5227.240259] LustreError: 172643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142765 with bad export cookie 6767159370124748226 [ 5227.246151] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.253075] LustreError: 172643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5227.608920] Lustre: server umount lustre-MDT0001 complete [ 5234.857899] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:26:12 (1783142772) [ 5250.745860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5259.223207] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5290.857588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.107575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5291.136987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5291.239828] Lustre: lustre-MDT0000: new disk, initializing [ 5291.315993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5291.321222] Lustre: Skipped 2 previous similar messages [ 5291.334761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5295.469596] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5304.458055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.588549] Lustre: 178951:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5304.887163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5309.006868] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5317.402438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5322.551561] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5328.804986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5329.213459] Lustre: 180927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5334.848762] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5337.156050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5337.253026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5342.862359] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5350.949552] Lustre: server umount lustre-OST0000 complete [ 5356.001316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5356.009520] LustreError: Skipped 1 previous similar message [ 5356.016901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5356.027311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5356.030925] Lustre: Skipped 8 previous similar messages [ 5364.305152] Lustre: server umount lustre-MDT0000 complete [ 5367.522577] LustreError: 178942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142905 with bad export cookie 6767159370124749311 [ 5367.524585] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5367.532961] LustreError: 178942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5377.165474] Lustre: server umount lustre-MDT0001 complete [ 5385.768968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5388.283102] Key type lgssc unregistered [ 5388.624768] LNet: 182783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5388.635149] LNetError: 182783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5388.666772] LNet: Removed LNI 192.168.204.146@tcp [ 5389.398802] Key type .llcrypt unregistered [ 5389.400900] Key type ._llcrypt unregistered [ 5408.891309] Key type ._llcrypt registered [ 5408.895852] Key type .llcrypt registered [ 5408.996539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5409.949145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5410.024962] alg: No test for adler32 (adler32-zlib) [ 5411.084047] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 5411.355381] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5412.995380] Key type lgssc registered [ 5413.980775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5420.988861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5421.003958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.399293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.126935] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5432.808326] LustreError: 184180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.829132] LustreError: 184180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5433.306521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.624969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5437.167868] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5442.556881] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.058391] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5452.088231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5452.319609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.934296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5462.085517] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5467.702086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5484.001215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5484.005709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.015483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.100323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5488.104967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5488.107159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.801696] Lustre: server umount lustre-OST0000 complete [ 5493.220355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.223268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5493.232557] Lustre: Skipped 1 previous similar message [ 5493.243223] Lustre: Skipped 1 previous similar message [ 5498.337870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5498.341806] Lustre: Skipped 2 previous similar messages [ 5499.061958] Lustre: server umount lustre-MDT0000 complete [ 5501.999877] LustreError: 184916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143040 with bad export cookie 9213099121136151237 [ 5502.013240] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5502.372758] Lustre: server umount lustre-MDT0001 complete [ 5511.152394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 5513.244325] Key type lgssc unregistered [ 5513.479092] LNet: 187736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5513.489394] LNetError: 187736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5513.506521] LNet: Removed LNI 192.168.204.146@tcp [ 5514.320489] Key type .llcrypt unregistered [ 5514.324197] Key type ._llcrypt unregistered [ 5533.910452] Key type ._llcrypt registered [ 5533.913349] Key type .llcrypt registered [ 5533.994756] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5534.885456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5534.915848] alg: No test for adler32 (adler32-zlib) [ 5535.924518] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 5536.111477] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 5537.744154] Key type lgssc registered [ 5538.770604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5545.212477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5545.229434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.654264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5550.279567] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5556.197755] LustreError: 189132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.207685] LustreError: 189132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5556.803508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.020399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5560.189257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5565.269584] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5569.228621] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5574.558956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5574.861583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.079971] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5584.085503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5591.281445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5600.863149] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5601.767292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.769643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.779304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.800955] Lustre: Skipped 1 previous similar message [ 5605.918620] Lustre: server umount lustre-OST0000 complete [ 5611.098828] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8947852fa800.ost_server_uuid 50 [ 5612.321826] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5627.361750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.367337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5627.378982] Lustre: Skipped 1 previous similar message [ 5627.388470] Lustre: Skipped 1 previous similar message [ 5630.526235] Lustre: server umount lustre-MDT0000 complete [ 5632.482347] LustreError: 189127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5632.495821] LustreError: 189127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5633.209245] LustreError: 189112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143171 with bad export cookie 16790954742112578259 [ 5633.210321] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5633.217184] LustreError: 189112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5633.452572] Lustre: server umount lustre-MDT0001 complete [ 5639.358573] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:32:57 (1783143177) [ 5643.748237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.056897] LustreError: 193066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5644.106024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.285785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5649.379902] LustreError: 193067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.463057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.771689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5657.015902] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5662.439491] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.908683] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5672.021504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5672.270678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.446137] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5681.408270] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5685.808652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5685.955611] Lustre: lustre-OST0001: new disk, initializing [ 5685.960743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5685.965682] Lustre: 195902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5689.137609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5690.528282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5694.001172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5694.008781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5694.111821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5698.067936] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5703.781475] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5703.791060] Lustre: Skipped 1 previous similar message [ 5704.161503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5704.168536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.880651] Lustre: lustre-OST0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 5708.885813] Lustre: Skipped 2 previous similar messages [ 5714.400300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5714.465759] Lustre: server umount lustre-OST0000 complete [ 5714.918151] LustreError: 194887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.927575] LustreError: 194887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5719.267353] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5719.428433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5723.008602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5723.184895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5724.243441] LustreError: 194885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.259643] LustreError: 194885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5740.517921] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5740.518974] LustreError: 194886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.521682] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.521690] Lustre: Skipped 1 previous similar message [ 5740.523730] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5740.523735] Lustre: Skipped 3 previous similar messages [ 5740.525279] LustreError: Skipped 2 previous similar messages [ 5740.564972] LustreError: 194886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5742.194726] Lustre: server umount lustre-OST0001 complete [ 5745.633475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.654043] Lustre: Skipped 1 previous similar message [ 5751.251365] Lustre: server umount lustre-MDT0000 complete [ 5754.268894] LustreError: 193047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143292 with bad export cookie 16790954742112579330 [ 5754.271912] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5754.282122] LustreError: 193047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5754.575610] Lustre: server umount lustre-MDT0001 complete [ 5773.380946] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:35:11 (1783143311) [ 5778.414232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.592491] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5778.610984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5778.735920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.741570] Lustre: Skipped 1 previous similar message [ 5782.429299] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5788.656163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.724483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5788.780180] Lustre: 199344:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5792.126687] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5797.386786] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5801.330788] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5805.946499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5806.131831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5810.428193] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5815.288412] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5816.388104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5819.798610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5819.846750] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5820.036547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5820.045572] Lustre: Skipped 2 previous similar messages [ 5824.127376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5829.085643] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5831.649068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5831.650857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5831.654248] LustreError: Skipped 1 previous similar message [ 5831.655531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5831.655537] Lustre: Skipped 5 previous similar messages [ 5831.675915] Lustre: Skipped 2 previous similar messages [ 5837.020700] Lustre: server umount lustre-OST0000 complete [ 5837.797200] LustreError: 202896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.811958] LustreError: 202896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5852.641411] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.646648] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5856.274996] Lustre: server umount lustre-OST0001 complete [ 5860.835561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5860.849116] LustreError: Skipped 1 previous similar message [ 5864.834641] Lustre: server umount lustre-MDT0000 complete [ 5867.225761] LustreError: 199335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143405 with bad export cookie 16790954742112580737 [ 5867.227684] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5867.234539] LustreError: 199335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5867.458715] Lustre: server umount lustre-MDT0001 complete [ 5883.900546] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:37:02 (1783143422) [ 5895.001388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 5901.437772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 5924.300842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.482730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5924.508084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5924.562252] Lustre: lustre-MDT0000: new disk, initializing [ 5924.628131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.650251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5927.445451] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5934.340541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.405548] Lustre: 207434:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5934.447292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5934.450477] Lustre: Skipped 1 previous similar message [ 5934.502830] Lustre: lustre-MDT0001: new disk, initializing [ 5934.605891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5934.620697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5937.182274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5943.182914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5946.443791] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5950.082587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5950.233918] Lustre: lustre-OST0000: new disk, initializing [ 5950.236870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5950.244253] Lustre: 209400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5953.529400] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5957.149505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5957.154545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5957.223594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5959.063135] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5961.537066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5961.665939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5964.040274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5964.159861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5967.333865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.338132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5967.338800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5967.338803] Lustre: Skipped 10 previous similar messages [ 5967.345830] Lustre: Skipped 4 previous similar messages [ 5967.351962] LustreError: Skipped 1 previous similar message [ 5971.412802] Lustre: server umount lustre-OST0000 complete [ 5979.842390] Lustre: server umount lustre-MDT0000 complete [ 5982.059163] LustreError: 210939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143520 with bad export cookie 16790954742112582298 [ 5982.063410] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.067072] LustreError: 210939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5982.276163] Lustre: server umount lustre-MDT0001 complete [ 5986.813986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.053229] LustreError: 211614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.063887] LustreError: 211614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5989.172682] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5993.719387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5996.096694] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 5999.835345] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6002.905980] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6006.610016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6010.134893] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6014.162817] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.800339] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6016.967165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6019.594670] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6019.728826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6022.115595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6022.119255] LustreError: Skipped 1 previous similar message [ 6027.003198] Lustre: server umount lustre-OST0000 complete [ 6030.966926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6033.823654] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6059.484766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6059.695889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.700957] Lustre: Skipped 5 previous similar messages [ 6062.934545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6064.810343] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.820661] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6066.945183] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6067.893157] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6070.019409] Lustre: server umount lustre-OST0000 complete [ 6072.142387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.144928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6072.149689] Lustre: Skipped 5 previous similar messages [ 6072.159688] Lustre: Skipped 8 previous similar messages [ 6078.319258] Lustre: server umount lustre-MDT0000 complete [ 6080.535602] LustreError: 211594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143618 with bad export cookie 16790954742112583180 [ 6080.536709] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6080.542963] LustreError: 211594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6090.814436] Lustre: server umount lustre-MDT0001 complete [ 6095.297479] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:40:33 (1783143633) [ 6098.859101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.496568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6106.186914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.590295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6112.591699] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6115.405618] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6119.154356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6122.578156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6126.167225] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6128.189940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6128.306691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6130.273326] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6132.466296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 6136.370200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.488640] Lustre: lustre-OST0001: new disk, initializing [ 6136.492096] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6136.495480] Lustre: 220952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6139.884378] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6145.493628] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6145.556672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6145.567298] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6145.629375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6148.067316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6148.176528] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6150.408800] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6150.543063] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6155.751800] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6155.758162] LustreError: Skipped 1 previous similar message [ 6157.651865] Lustre: server umount lustre-OST0001 complete [ 6160.868455] LustreError: 220570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.878179] LustreError: 220570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6161.135916] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6161.252987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6163.776181] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6163.906924] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6189.457894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.805510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6194.663777] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6194.670949] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6194.671741] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6196.924260] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6197.914580] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6200.410933] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 6200.416832] Lustre: Skipped 6 previous similar messages [ 6205.131878] Lustre: server umount lustre-OST0001 complete [ 6220.260524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.268290] Lustre: Skipped 4 previous similar messages [ 6233.056818] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6233.158752] Lustre: server umount lustre-OST0000 complete [ 6241.467061] Lustre: server umount lustre-MDT0000 complete [ 6243.372163] LustreError: 220951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143781 with bad export cookie 16790954742112584125 [ 6243.377921] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6243.379224] LustreError: 220951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6249.642955] Lustre: server umount lustre-MDT0001 complete [ 6262.288833] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:43:20 (1783143800) [ 6265.765656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.911250] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6265.929171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6268.219820] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6273.017542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.086226] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6273.099889] Lustre: 226188:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6275.206131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6278.639808] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6281.307161] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6284.699238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.821932] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6287.657894] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6291.310468] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6296.047849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.092039] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6299.018156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6302.682837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6303.504253] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6305.810436] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6305.930832] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6308.019219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6308.132818] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6310.842849] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894784fc7000.ost_server_uuid 50 [ 6311.652451] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff894784fc7000.ost_server_uuid in FULL state after 0 sec [ 6312.219865] Lustre: Permanently deactivating lustre-OST0001 [ 6312.229230] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6313.920236] Lustre: Permanently reactivating lustre-OST0001 [ 6317.539138] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6317.545273] LustreError: Skipped 4 previous similar messages [ 6321.936977] Lustre: server umount lustre-OST0001 complete [ 6330.028740] Lustre: server umount lustre-OST0000 complete [ 6338.149146] LustreError: 231156:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000bbe18217) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6338.200484] Lustre: server umount lustre-MDT0000 complete [ 6340.075269] LustreError: 226180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143878 with bad export cookie 16790954742112585413 [ 6340.078611] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6340.080882] LustreError: 226180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6340.267531] Lustre: server umount lustre-MDT0001 complete [ 6352.444321] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:44:50 (1783143890) [ 6356.559098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.722329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6356.742873] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6356.748598] Lustre: Skipped 5 previous similar messages [ 6356.857235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6356.864796] Lustre: Skipped 9 previous similar messages [ 6359.172032] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6364.035177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6364.091945] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6364.107621] Lustre: 232876:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6366.435591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6370.311345] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6373.272795] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6377.089023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6377.250070] Lustre: Permanently deactivating lustre-OST0000 [ 6377.270514] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6380.162440] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6381.146219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6385.364645] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6389.010569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.379487] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6396.154331] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6398.522456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6402.230096] Lustre: Permanently reactivating lustre-OST0000 [ 6402.240545] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6402.247111] Lustre: Skipped 2 previous similar messages [ 6407.691755] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6407.700563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.711034] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6407.717977] Lustre: Skipped 1 previous similar message [ 6407.740253] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6407.755395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6416.269958] Lustre: server umount lustre-OST0001 complete [ 6418.914982] LustreError: 236420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6418.923745] LustreError: 236420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 6424.235630] Lustre: server umount lustre-OST0000 complete [ 6432.389110] Lustre: server umount lustre-MDT0000 complete [ 6434.359718] LustreError: 232867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143972 with bad export cookie 16790954742112587065 [ 6434.363801] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6434.369823] LustreError: 232867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6434.592846] Lustre: server umount lustre-MDT0001 complete [ 6440.194331] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6441.720511] Key type lgssc unregistered [ 6441.893536] LNet: 237732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6441.902574] LNetError: 237732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6442.919510] LNet: Removed LNI 192.168.204.146@tcp [ 6443.389052] Key type .llcrypt unregistered [ 6443.396859] Key type ._llcrypt unregistered [ 6455.449983] Key type ._llcrypt registered [ 6455.451839] Key type .llcrypt registered [ 6455.507899] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6455.965374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6456.025287] alg: No test for adler32 (adler32-zlib) [ 6456.945753] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 6457.077560] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6458.696566] Key type lgssc registered [ 6459.281756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6464.545563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6464.561176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.805772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.857866] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6472.262165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.328727] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6472.348763] Lustre: 239167:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6472.357610] Lustre: Permanently deactivating lustre-MDT0001 [ 6472.368252] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6472.514916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6474.711108] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6475.386077] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6475.390301] LustreError: 239210:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6475.399399] LustreError: 239210:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6475.407285] LustreError: 239210:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 6478.386448] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6481.194402] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6485.138041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.344207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6488.921618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6490.475021] LustreError: 241019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.480667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6490.489149] LustreError: 241019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6492.711232] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6495.716680] LustreError: 241016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.532892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.653533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6497.712843] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6497.723145] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6499.765414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6503.607088] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6506.775201] Lustre: Permanently reactivating lustre-MDT0001 [ 6506.785312] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6506.788566] Lustre: Skipped 2 previous similar messages [ 6512.739844] Lustre: lustre-MDT0001: Client ea3144e2-e506-4463-94a8-b0473e277f6b (at 192.168.204.46@tcp) reconnecting [ 6516.242866] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6517.186917] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6519.898752] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6520.021633] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6522.227538] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6522.340291] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6523.942883] Lustre: Permanently deactivating lustre-MDT0001 [ 6523.956904] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6523.962717] Lustre: Skipped 2 previous similar messages [ 6532.128177] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6538.727775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.731946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6538.744211] Lustre: Skipped 3 previous similar messages [ 6538.749864] Lustre: Skipped 2 previous similar messages [ 6542.200020] Lustre: server umount lustre-MDT0000 complete [ 6543.843038] LustreError: 239937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.851348] LustreError: 239937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6544.130217] LustreError: 239161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144082 with bad export cookie 15150610234242396280 [ 6544.132298] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.138447] LustreError: 239161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6544.290509] Lustre: server umount lustre-MDT0001 complete [ 6552.260152] Lustre: server umount lustre-OST0000 complete [ 6557.909929] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:48:16 (1783144096) [ 6563.016954] Lustre: server umount lustre-OST0001 complete [ 6565.901785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6570.296329] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6587.358041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6587.480862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6587.486180] Lustre: Skipped 2 previous similar messages [ 6587.502598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6587.543249] Lustre: lustre-MDT0000: new disk, initializing [ 6587.581465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6587.590049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6589.415581] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6594.906537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.958303] Lustre: 247425:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6594.990179] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6594.992980] Lustre: Skipped 1 previous similar message [ 6595.034277] Lustre: lustre-MDT0001: new disk, initializing [ 6595.095904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6595.102442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6596.992172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6601.821436] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6604.154013] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6607.304685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.476314] Lustre: lustre-OST0000: new disk, initializing [ 6607.478725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6607.482816] Lustre: 249391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6607.539762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.546208] Lustre: Skipped 1 previous similar message [ 6610.233885] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6615.409560] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6616.602278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6616.613142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6616.672351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6621.641614] LustreError: 247430:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6627.253714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.347453] LustreError: 250109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6627.359431] Lustre: lustre-OST0001: new disk, initializing [ 6627.365196] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6627.371344] Lustre: 250508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6627.426529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6630.503099] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6631.672136] LustreError: 247430:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6631.685557] LustreError: 247432:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6635.039798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6635.047157] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6635.109202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6635.780334] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6636.632370] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6641.784116] LustreError: 247432:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6642.784597] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 6642.789281] Lustre: Skipped 1 previous similar message [ 6645.218815] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6645.222663] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.226731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6645.228824] Lustre: Skipped 1 previous similar message [ 6647.888987] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 6647.895566] Lustre: Skipped 1 previous similar message [ 6653.004474] Lustre: lustre-OST0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 6653.009051] Lustre: Skipped 2 previous similar messages [ 6656.992721] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6657.067542] Lustre: server umount lustre-OST0001 complete [ 6658.122905] LustreError: 249374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.819660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.822061] LustreError: 249375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.823613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6670.823620] Lustre: Skipped 2 previous similar messages [ 6670.827958] Lustre: Skipped 1 previous similar message [ 6670.838233] LustreError: 249375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6676.418035] Lustre: server umount lustre-OST0000 complete [ 6681.059540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.074464] Lustre: Skipped 1 previous similar message [ 6684.619374] Lustre: server umount lustre-MDT0000 complete [ 6686.618140] LustreError: 247417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144224 with bad export cookie 15150610234242398870 [ 6686.621618] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.624462] LustreError: 247417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6692.827235] Lustre: server umount lustre-MDT0001 complete [ 6697.828365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6699.322189] Key type lgssc unregistered [ 6699.477472] LNet: 252539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.481558] LNetError: 252539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6699.490461] LNet: Removed LNI 192.168.204.146@tcp [ 6699.901215] Key type .llcrypt unregistered [ 6699.904070] Key type ._llcrypt unregistered [ 6720.414767] Key type ._llcrypt registered [ 6720.416281] Key type .llcrypt registered [ 6720.474718] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6721.013412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6721.042577] alg: No test for adler32 (adler32-zlib) [ 6721.964777] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 6722.084659] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6723.696204] Key type lgssc registered [ 6724.258452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6728.131071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6728.142855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.275360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6729.287111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6729.365225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.239360] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6735.794293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.870906] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6735.898206] Lustre: 254725:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6736.080255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6738.382384] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6742.476332] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6745.858783] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6749.948475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6750.051949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6750.156502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6753.348104] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6756.406689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6756.990496] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6802.401826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.402719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.407852] LustreError: Skipped 1 previous similar message [ 6802.410180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.413932] Lustre: Skipped 1 previous similar message [ 6805.663297] Lustre: server umount lustre-OST0000 complete [ 6808.026049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6814.319576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.526418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.462707] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6820.891958] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6822.700988] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6822.710193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6822.713944] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6822.720943] Lustre: Skipped 1 previous similar message [ 6858.722517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6858.724120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6858.724788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6858.724792] Lustre: Skipped 1 previous similar message [ 6858.728820] Lustre: Skipped 1 previous similar message [ 6858.731556] LustreError: Skipped 1 previous similar message [ 6862.009910] Lustre: server umount lustre-OST0000 complete [ 6863.841472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.842187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6863.850700] Lustre: Skipped 1 previous similar message [ 6863.853391] Lustre: Skipped 1 previous similar message [ 6868.961136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6868.963977] Lustre: Skipped 2 previous similar messages [ 6874.081068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6874.083673] Lustre: Skipped 1 previous similar message [ 6878.176149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6878.260804] Lustre: server umount lustre-MDT0000 complete [ 6879.868682] LustreError: 254716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144418 with bad export cookie 17531964408514288065 [ 6879.869562] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6879.874404] LustreError: 254716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6880.009790] Lustre: server umount lustre-MDT0001 complete [ 6884.775785] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6886.071174] Key type lgssc unregistered [ 6886.217589] LNet: 260142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6886.223121] LNetError: 260142:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6886.236477] LNet: Removed LNI 192.168.204.146@tcp [ 6886.614546] Key type .llcrypt unregistered [ 6886.616423] Key type ._llcrypt unregistered [ 6897.985704] Key type ._llcrypt registered [ 6897.987453] Key type .llcrypt registered [ 6898.042632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6898.470036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6898.483199] alg: No test for adler32 (adler32-zlib) [ 6899.383622] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 6899.503033] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6901.120396] Key type lgssc registered [ 6901.673138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6905.221316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6905.230046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.447812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.171504] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6911.406384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.543252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.244475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6916.118394] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6918.546984] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6921.345879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6921.497342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6924.211647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6927.500735] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6932.779380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6943.202547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6943.204808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6943.206037] LustreError: Skipped 1 previous similar message [ 6943.215074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6943.219679] Lustre: Skipped 1 previous similar message [ 6944.150389] Lustre: server umount lustre-OST0000 complete [ 6947.297134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6947.301141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6947.307262] Lustre: Skipped 1 previous similar message [ 6947.310116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6948.004701] Lustre: server umount lustre-MDT0000 complete [ 6948.321095] LustreError: 261537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.321729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.328273] LustreError: 261537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6949.413600] LustreError: 262271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144487 with bad export cookie 15803480027331818390 [ 6949.417774] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6949.420884] LustreError: 262271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6949.541105] Lustre: server umount lustre-MDT0001 complete [ 6953.962976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6955.283368] Key type lgssc unregistered [ 6955.434838] LNet: 265838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6955.439201] LNetError: 265838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6955.448544] LNet: Removed LNI 192.168.204.146@tcp [ 6955.795219] Key type .llcrypt unregistered [ 6955.797787] Key type ._llcrypt unregistered [ 6963.661532] Key type ._llcrypt registered [ 6963.663283] Key type .llcrypt registered [ 6963.716565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6964.143974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.166738] alg: No test for adler32 (adler32-zlib) [ 6965.039345] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 6965.153474] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6966.760252] Key type lgssc registered [ 6967.290921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6970.510883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6970.519857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.715691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.539852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6977.428749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.613910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6979.567446] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6982.803742] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6985.164351] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6988.258637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.444946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.908518] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6993.925219] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6996.649870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7011.811383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7011.817141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.827205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.347633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.347764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7013.356374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.468653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7018.472035] Lustre: Skipped 2 previous similar messages [ 7023.072168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7023.163973] Lustre: server umount lustre-OST0000 complete [ 7028.708536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.709060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7028.710881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7028.715114] Lustre: Skipped 1 previous similar message [ 7031.266016] Lustre: server umount lustre-MDT0000 complete [ 7033.031417] LustreError: 267018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144571 with bad export cookie 10034490169934490138 [ 7033.036278] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7033.040954] LustreError: 267018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7033.206372] Lustre: server umount lustre-MDT0001 complete [ 7038.432851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7040.011234] Key type lgssc unregistered [ 7040.200559] LNet: 270732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7040.207677] LNetError: 270732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7040.219578] LNet: Removed LNI 192.168.204.146@tcp [ 7040.725869] Key type .llcrypt unregistered [ 7040.728702] Key type ._llcrypt unregistered [ 7053.891228] Key type ._llcrypt registered [ 7053.892552] Key type .llcrypt registered [ 7053.938776] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7054.345527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7054.358878] alg: No test for adler32 (adler32-zlib) [ 7055.290455] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7055.418324] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7057.016429] Key type lgssc registered [ 7057.644371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7061.745820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7061.754237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7062.980660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7065.258779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7069.538689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7069.722535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.588213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7074.915131] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7077.402660] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7080.774867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7080.958671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.970159] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7087.468254] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7088.112291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7108.577557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7108.581187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.588605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7109.602606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7109.604470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7109.607457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.762907] Lustre: server umount lustre-OST0000 complete [ 7115.746793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7115.751595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7115.759209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.765416] Lustre: Skipped 1 previous similar message [ 7119.842483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.843984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7119.859039] Lustre: Skipped 1 previous similar message [ 7121.773197] Lustre: server umount lustre-MDT0000 complete [ 7123.577282] LustreError: 272108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144661 with bad export cookie 14862680172397046898 [ 7123.582048] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7123.585357] LustreError: 272108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7123.751432] Lustre: server umount lustre-MDT0001 complete [ 7128.939947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7130.311094] Key type lgssc unregistered [ 7130.486826] LNet: 276448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7130.495378] LNetError: 276448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7130.511757] LNet: Removed LNI 192.168.204.146@tcp [ 7130.921981] Key type .llcrypt unregistered [ 7130.924360] Key type ._llcrypt unregistered [ 7139.679543] Key type ._llcrypt registered [ 7139.680983] Key type .llcrypt registered [ 7139.728032] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7140.196840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7140.224593] alg: No test for adler32 (adler32-zlib) [ 7141.105902] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7141.208912] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7142.800361] Key type lgssc registered [ 7143.296091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7146.754994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7146.767699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7148.036513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.808387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7153.248987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.413606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7155.233864] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7158.364551] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7160.554907] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7163.389848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.589154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7165.910199] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7168.906797] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7173.869793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7179.233226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7179.236196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.239131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.241407] Lustre: Skipped 1 previous similar message [ 7179.246039] Lustre: Skipped 1 previous similar message [ 7180.003380] Lustre: server umount lustre-OST0000 complete [ 7183.517776] Lustre: server umount lustre-MDT0000 complete [ 7184.352939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7184.354949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7184.356730] LustreError: 277662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.361268] Lustre: Skipped 1 previous similar message [ 7184.369955] LustreError: 277662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7185.038351] LustreError: 277626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144723 with bad export cookie 11709895257583737947 [ 7185.041707] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7185.046649] LustreError: 277626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7185.179661] Lustre: server umount lustre-MDT0001 complete [ 7189.620066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7190.944864] Key type lgssc unregistered [ 7191.097484] LNet: 281355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7191.102239] LNetError: 281355:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7191.119509] LNet: Removed LNI 192.168.204.146@tcp [ 7191.501335] Key type .llcrypt unregistered [ 7191.503062] Key type ._llcrypt unregistered [ 7196.391647] Key type ._llcrypt registered [ 7196.393385] Key type .llcrypt registered [ 7196.448402] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7204.161879] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7210.858278] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7211.292712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7211.339371] alg: No test for adler32 (adler32-zlib) [ 7212.249251] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7212.370745] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7213.960276] Key type lgssc registered [ 7214.394281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7227.918665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7227.925817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7229.018141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7229.036373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7229.078133] Lustre: lustre-MDT0000: new disk, initializing [ 7229.123973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.132680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7230.767921] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7235.281329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.322899] Lustre: 284879:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7235.348629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7235.350897] Lustre: Skipped 1 previous similar message [ 7235.389383] Lustre: lustre-MDT0001: new disk, initializing [ 7235.424522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.439783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7235.449515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7236.964925] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7241.074557] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7243.254533] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7245.988788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7246.101620] Lustre: lustre-OST0000: new disk, initializing [ 7246.104171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7246.107700] Lustre: 286847:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7246.140354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.549353] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7252.869799] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7254.583125] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7256.065860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7256.070788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7256.123565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7256.739846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7258.447459] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7258.542614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7261.156269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7261.156322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.158693] LustreError: Skipped 1 previous similar message [ 7261.163435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7261.167429] Lustre: Skipped 1 previous similar message [ 7265.440228] Lustre: server umount lustre-OST0000 complete [ 7271.393757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7271.393844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7271.394722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7271.407216] Lustre: Skipped 2 previous similar messages [ 7276.516037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7276.522788] Lustre: Skipped 2 previous similar messages [ 7281.120189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7281.198206] Lustre: server umount lustre-MDT0000 complete [ 7282.755027] LustreError: 285674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144820 with bad export cookie 7292732819443197265 [ 7282.760013] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7282.761599] LustreError: 285674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7282.899980] Lustre: server umount lustre-MDT0001 complete [ 7286.435509] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:00:25 (1783144825) [ 7289.058058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.274053] LustreError: 289271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.282885] LustreError: 289271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7289.301300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7290.818332] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7294.039240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.178495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7295.718816] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7298.459128] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7300.614199] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7303.214866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.351718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.713540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7308.174906] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7325.154482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.154996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.164545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.167482] Lustre: Skipped 1 previous similar message [ 7327.832226] Lustre: server umount lustre-OST0000 complete [ 7330.273807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7330.274306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.276843] LustreError: Skipped 1 previous similar message [ 7330.277658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7330.283171] Lustre: Skipped 2 previous similar messages [ 7330.289898] Lustre: Skipped 1 previous similar message [ 7335.527060] Lustre: server umount lustre-MDT0000 complete [ 7337.110462] LustreError: 291111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144875 with bad export cookie 7292732819443198133 [ 7337.110739] LustreError: 290007:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 7337.111573] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7337.117035] LustreError: 291111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7337.245995] Lustre: server umount lustre-MDT0001 complete [ 7341.519852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7342.817885] Key type lgssc unregistered [ 7342.956673] LNet: 292834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7342.960405] LNetError: 292834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7342.970626] LNet: Removed LNI 192.168.204.146@tcp [ 7343.433619] Key type .llcrypt unregistered [ 7343.435613] Key type ._llcrypt unregistered [ 7354.754551] Key type ._llcrypt registered [ 7354.755984] Key type .llcrypt registered [ 7354.806414] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7361.914525] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7362.359564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.369854] alg: No test for adler32 (adler32-zlib) [ 7363.241314] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7363.343681] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7364.928164] Key type lgssc registered [ 7365.383214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7384.393614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7384.404822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.503892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7385.517867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7385.555189] Lustre: lustre-MDT0000: new disk, initializing [ 7385.586339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.594294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7387.222541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7391.763467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.811739] Lustre: 296634:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7391.839612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7391.842225] Lustre: Skipped 1 previous similar message [ 7391.881377] Lustre: lustre-MDT0001: new disk, initializing [ 7391.924400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7391.941598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7391.946810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7393.417425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7397.485149] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7399.599115] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7402.488382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7402.592896] Lustre: lustre-OST2710: new disk, initializing [ 7402.594792] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7402.598152] Lustre: 298601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7402.637696] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7404.881969] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7406.529561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7406.533824] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7406.561904] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7408.793235] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7411.381851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.454272] Lustre: lustre-OST03e8: new disk, initializing [ 7411.456715] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7411.459357] Lustre: 299667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7411.495162] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7413.611595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7415.816073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7415.819807] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7415.853931] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7416.339424] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7419.393393] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7419.480404] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7421.169201] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7421.266429] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7423.280556] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7423.383137] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7425.131728] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7425.220956] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7428.065209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7428.068876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7428.074920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7431.141345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7431.142516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7431.152418] Lustre: Skipped 1 previous similar message [ 7431.155400] Lustre: Skipped 1 previous similar message [ 7432.552710] Lustre: server umount lustre-MDT0000 complete [ 7434.028200] LustreError: 296625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144972 with bad export cookie 14730866491473893080 [ 7434.031909] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7434.033552] LustreError: 296625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7434.166896] Lustre: server umount lustre-MDT0001 complete [ 7445.455729] Lustre: server umount lustre-OST2710 complete [ 7457.383668] Lustre: server umount lustre-OST03e8 complete [ 7465.477994] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7469.958887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7484.922738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.035693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7485.051680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7485.100655] Lustre: lustre-MDT0000: new disk, initializing [ 7485.137262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.146529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7486.764698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7491.352576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.400895] Lustre: 304921:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7491.513223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7492.956807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7496.914820] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7498.952526] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7501.621830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7501.747778] Lustre: lustre-OST0000: new disk, initializing [ 7501.749892] Lustre: Skipped 1 previous similar message [ 7501.752454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7501.754785] Lustre: Skipped 2 previous similar messages [ 7501.757867] Lustre: 306891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7501.799617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.803597] Lustre: Skipped 1 previous similar message [ 7504.186779] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7508.628334] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7510.196610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7511.545591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7511.548929] Lustre: Skipped 1 previous similar message [ 7511.550976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7511.584951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7512.331564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 7514.079602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7514.186587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7516.642812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7516.642935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.643581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.645916] LustreError: Skipped 1 previous similar message [ 7516.650311] Lustre: Skipped 2 previous similar messages [ 7521.048118] Lustre: server umount lustre-OST0000 complete [ 7526.881678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.882180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7526.887065] Lustre: Skipped 1 previous similar message [ 7526.889291] Lustre: Skipped 1 previous similar message [ 7528.547949] Lustre: server umount lustre-MDT0000 complete [ 7529.908235] LustreError: 304914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145068 with bad export cookie 14730866491473894760 [ 7529.911757] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7529.914032] LustreError: 304914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7530.045322] Lustre: server umount lustre-MDT0001 complete [ 7533.231511] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:04:31 (1783145071) [ 7533.790691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7534.397475] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:04:33 (1783145073) [ 7535.568220] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7545.112261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.214758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7545.223850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7545.294691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.835813] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7549.996035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.041984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7550.054051] Lustre: 310758:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7551.681529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7568.100234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7568.190703] Lustre: Denying initial registration attempt from nid 192.168.204.146@tcp, specified as failover [ 7568.195167] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7568.200234] LustreError: 312209:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 7568.203727] LustreError: 312209:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7568.206734] LustreError: 312209:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7568.237704] Lustre: server umount lustre-OST0000 complete [ 7568.239706] LustreError: 312209:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7570.913323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7570.916862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7570.923595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.926321] Lustre: Skipped 1 previous similar message [ 7581.665745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7581.668874] Lustre: Skipped 5 previous similar messages [ 7585.248166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7585.318760] Lustre: server umount lustre-MDT0000 complete [ 7586.894255] LustreError: 311499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145125 with bad export cookie 14730866491473895628 [ 7586.897614] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7586.902599] LustreError: 311499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7587.056264] Lustre: server umount lustre-MDT0001 complete [ 7591.763462] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7592.975722] Key type lgssc unregistered [ 7593.113390] LNet: 313331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7593.118402] LNetError: 313331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7593.129444] LNet: Removed LNI 192.168.204.146@tcp [ 7593.480589] Key type .llcrypt unregistered [ 7593.482368] Key type ._llcrypt unregistered [ 7597.824024] Key type ._llcrypt registered [ 7597.825579] Key type .llcrypt registered [ 7597.872228] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7598.226696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7598.273030] alg: No test for adler32 (adler32-zlib) [ 7599.140345] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7599.266861] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7600.864177] Key type lgssc registered [ 7601.324488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.620460] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7631.129889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7631.138557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7632.247924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7632.264360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7632.340194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.024884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7637.401638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7637.453346] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7637.466186] Lustre: 316071:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7637.579196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7639.184587] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7642.206289] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7644.343870] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7660.133673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.220757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7660.310566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7662.729115] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7665.619613] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7666.507448] Lustre: server umount lustre-OST0000 complete [ 7668.193087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7668.196091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.201805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.777957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.778360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.151078] Lustre: server umount lustre-MDT0000 complete [ 7675.728613] LustreError: 317934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145213 with bad export cookie 18002995536328012114 [ 7675.732253] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7675.734795] LustreError: 317934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7675.875211] Lustre: server umount lustre-MDT0001 complete [ 7680.144913] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7681.271895] Key type lgssc unregistered [ 7681.408550] LNet: 319638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7681.412428] LNetError: 319638:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7681.419402] LNet: Removed LNI 192.168.204.146@tcp [ 7681.780442] Key type .llcrypt unregistered [ 7681.781812] Key type ._llcrypt unregistered [ 7692.633558] Key type ._llcrypt registered [ 7692.634967] Key type .llcrypt registered [ 7692.679833] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7693.088247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7693.162383] alg: No test for adler32 (adler32-zlib) [ 7694.026256] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7694.137441] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7695.728153] Key type lgssc registered [ 7696.188872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7699.125958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7699.133392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.298078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.837607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7704.953153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7705.105748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.784255] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7709.626720] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7711.797446] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7716.644981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7716.782681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7718.965457] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7721.546418] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7735.779257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7735.782741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.788733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7738.456481] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 7740.387445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7740.389577] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.395513] Lustre: Skipped 1 previous similar message [ 7741.853495] Lustre: server umount lustre-MDT0000 complete [ 7743.300583] LustreError: 321058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145281 with bad export cookie 15976995754007623399 [ 7743.305020] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.305917] LustreError: 321058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7743.561326] LustreError: 321028:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.569469] LustreError: 321028:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7743.570167] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 7743.575959] Lustre: Skipped 2 previous similar messages [ 7745.504841] LustreError: 321027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.505129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7748.513531] LustreError: 321027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.513581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7748.522952] Lustre: Skipped 1 previous similar message [ 7749.505701] Lustre: server umount lustre-MDT0001 complete [ 7752.101952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7755.716208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.032977] LustreError: 320343:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000aa9752a8 x1869763294814976/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7768.196903] LustreError: 322862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.209394] LustreError: 322862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7768.236308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7768.270500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7770.032336] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7773.345143] LustreError: 324641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.351706] LustreError: 324641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7773.461039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.660650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7775.204257] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7778.062331] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7778.794825] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7780.805056] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 7782.811985] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7783.392808] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7789.027345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.030205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.031279] LustreError: Skipped 1 previous similar message [ 7789.040613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.044891] Lustre: Skipped 2 previous similar messages [ 7799.264130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7799.331323] Lustre: server umount lustre-OST0000 complete [ 7804.386322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.387229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7804.392033] Lustre: Skipped 1 previous similar message [ 7806.742647] Lustre: server umount lustre-MDT0000 complete [ 7807.956143] LustreError: 321014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145346 with bad export cookie 15976995754007630553 [ 7807.959623] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7807.960549] LustreError: 321014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7808.066499] Lustre: server umount lustre-MDT0001 complete [ 7811.719674] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7812.752125] Key type lgssc unregistered [ 7812.882376] LNet: 327313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7812.885074] LNetError: 327313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7812.891262] LNet: Removed LNI 192.168.204.146@tcp [ 7813.216893] Key type .llcrypt unregistered [ 7813.217977] Key type ._llcrypt unregistered [ 7822.826914] Key type ._llcrypt registered [ 7822.828338] Key type .llcrypt registered [ 7822.869097] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7823.198407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7823.264338] alg: No test for adler32 (adler32-zlib) [ 7824.138440] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7824.226101] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7825.808183] Key type lgssc registered [ 7826.177897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7828.958126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7828.965421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.142622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7831.610161] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7834.540416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.672622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7836.151205] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7838.765734] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7840.627978] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7844.833506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7844.834350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7844.838291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7844.841687] Lustre: Skipped 1 previous similar message [ 7844.845011] Lustre: Skipped 1 previous similar message [ 7849.953740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7849.957410] Lustre: Skipped 1 previous similar message [ 7855.073475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7855.077931] Lustre: Skipped 1 previous similar message [ 7856.096156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7856.164606] Lustre: server umount lustre-MDT0000 complete [ 7857.615583] LustreError: 328686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145395 with bad export cookie 12020389966801732806 [ 7857.619334] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7857.620277] LustreError: 328686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7857.730200] Lustre: server umount lustre-MDT0001 complete [ 7863.306173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7863.419651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7863.434687] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7863.526105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.136666] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7868.532659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.573031] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7868.586400] Lustre: 331292:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7868.709682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7870.328432] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7873.118802] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7875.222891] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7880.227792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7880.323596] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7880.330222] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7880.334464] LustreError: 333175:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7880.338282] LustreError: 333175:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7880.341731] LustreError: 333175:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7880.369104] Lustre: server umount lustre-OST0000 complete [ 7880.370709] LustreError: 333175:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7885.797220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7885.881255] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7885.959511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7888.226401] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7890.775057] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7893.574792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.613743] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7893.646275] Lustre: lustre-OST0001: new disk, initializing [ 7893.648696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7893.652207] Lustre: 334703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7895.832068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7895.835507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7895.864307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7895.888523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7895.900763] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7898.542847] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7901.156149] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.159402] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7901.163658] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7901.167274] LustreError: Skipped 1 previous similar message [ 7901.170043] Lustre: Skipped 1 previous similar message [ 7905.736042] Lustre: server umount lustre-OST0001 complete [ 7906.276251] LustreError: 333641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.282532] LustreError: 333641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7911.397466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7911.398518] LustreError: 335042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7911.398902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7911.398914] Lustre: Skipped 1 previous similar message [ 7911.401964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7911.401983] LustreError: Skipped 1 previous similar message [ 7911.409237] LustreError: 335042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7913.296230] Lustre: server umount lustre-OST0000 complete [ 7914.977045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7914.980914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7914.985953] Lustre: Skipped 1 previous similar message [ 7920.813723] Lustre: server umount lustre-MDT0000 complete [ 7921.633083] LustreError: 332048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.641247] LustreError: 332048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7922.242424] LustreError: 331282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145460 with bad export cookie 12020389966801733345 [ 7922.247287] LustreError: 331282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7922.247333] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.379330] Lustre: server umount lustre-MDT0001 complete [ 7926.302352] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7927.404886] Key type lgssc unregistered [ 7927.535383] LNet: 336665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.538970] LNetError: 336665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.546516] LNet: Removed LNI 192.168.204.146@tcp [ 7927.856277] Key type .llcrypt unregistered [ 7927.857850] Key type ._llcrypt unregistered [ 7960.028268] Key type ._llcrypt registered [ 7960.029877] Key type .llcrypt registered [ 7960.069934] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7966.284887] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7966.695627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7966.708261] alg: No test for adler32 (adler32-zlib) [ 7967.562792] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 7967.657637] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7969.240182] Key type lgssc registered [ 7969.649491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7981.579472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7981.588076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.668685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.678828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7982.715973] Lustre: lustre-MDT0000: new disk, initializing [ 7982.737966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.745652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7984.070035] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7988.498775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.545469] Lustre: 341566:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7988.570795] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7988.573409] Lustre: Skipped 1 previous similar message [ 7988.610987] Lustre: lustre-MDT0001: new disk, initializing [ 7988.646077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7988.661057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7988.664494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7990.072590] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7993.836992] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7995.753124] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7998.339020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.447894] Lustre: lustre-OST0000: new disk, initializing [ 7998.449804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7998.452601] Lustre: 343535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7998.482732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.382627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8000.386794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8000.424046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8000.737163] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8004.864083] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8006.378197] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8006.453048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8008.116828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8008.212562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8010.723781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8010.725200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.727012] LustreError: Skipped 1 previous similar message [ 8010.733930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8010.738101] Lustre: Skipped 1 previous similar message [ 8015.132411] Lustre: server umount lustre-OST0000 complete [ 8019.425453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8019.429472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.435143] Lustre: Skipped 1 previous similar message [ 8019.438083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.961230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.961796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8022.625926] Lustre: server umount lustre-MDT0000 complete [ 8024.083522] LustreError: 341604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145562 with bad export cookie 7128849056663643990 [ 8024.087158] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.088114] LustreError: 341604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8024.224634] Lustre: server umount lustre-MDT0001 complete [ 8027.439799] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:12:46 (1783145566) [ 8030.279321] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:12:48 (1783145568) [ 8033.547839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.786649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.285759] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8038.300986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8038.427824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8039.862826] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8042.485848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8044.350251] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8046.710281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8046.823510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.852863] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8051.404173] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8065.662213] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8066.407554] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8067.119808] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8067.866835] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8068.616353] Lustre: Failing over lustre-MDT0000 [ 8068.804859] Lustre: server umount lustre-MDT0000 complete [ 8069.089190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8069.092547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8069.098512] LustreError: 347681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.103904] LustreError: 347681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8069.603381] LustreError: 348771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.610903] LustreError: 348771:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8072.275563] LustreError: 348771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8074.721332] LustreError: 346287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8074.727568] LustreError: 346287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8079.843179] LustreError: 346286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8079.849103] LustreError: 346286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8082.305249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.377315] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8082.499574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.523533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8084.017706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8087.521621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8087.526580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8087.640938] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8087.662446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8090.065433] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8091.040386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8093.120907] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8093.957078] Lustre: Failing over lustre-MDT0000 [ 8094.187877] Lustre: server umount lustre-MDT0000 complete [ 8097.762499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8097.763764] LustreError: 347681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.766890] Lustre: Skipped 4 previous similar messages [ 8097.779363] LustreError: 347681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8107.863249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.907756] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.034821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8108.106473] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8109.484761] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8113.125588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8113.127685] Lustre: Skipped 2 previous similar messages [ 8113.140303] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8113.158968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8115.253301] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8115.855901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8123.361625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8123.362325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8123.363898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8123.363904] Lustre: Skipped 1 previous similar message [ 8123.372951] Lustre: Skipped 2 previous similar messages [ 8124.444531] Lustre: server umount lustre-MDT0000 complete [ 8125.743543] LustreError: 346268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145663 with bad export cookie 7128849056663646671 [ 8125.746096] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.748104] LustreError: 346268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8125.863206] Lustre: server umount lustre-MDT0001 complete [ 8137.700922] Lustre: server umount lustre-OST0000 complete [ 8143.414976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8143.563796] LustreError: 352634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.571365] LustreError: 352634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8143.590400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8143.595664] Lustre: Skipped 1 previous similar message [ 8145.086583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8148.145799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.637910] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8152.358247] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8154.330141] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8156.786827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.900550] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8161.462587] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8168.167357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8178.659392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.661955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8178.665408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8178.667957] LustreError: Skipped 1 previous similar message [ 8178.670432] Lustre: Skipped 3 previous similar messages [ 8179.667905] Lustre: server umount lustre-OST0000 complete [ 8187.162748] Lustre: server umount lustre-MDT0000 complete [ 8188.583504] LustreError: 352615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145726 with bad export cookie 7128849056663647420 [ 8188.585670] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.588481] LustreError: 352615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8188.719314] Lustre: server umount lustre-MDT0001 complete [ 8192.546301] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8193.616457] Key type lgssc unregistered [ 8193.748478] LNet: 356185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8193.750627] LNetError: 356185:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8193.759346] LNet: Removed LNI 192.168.204.146@tcp [ 8194.090209] Key type .llcrypt unregistered [ 8194.092150] Key type ._llcrypt unregistered [ 8204.406648] Key type ._llcrypt registered [ 8204.408326] Key type .llcrypt registered [ 8204.447121] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8204.814271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8204.838668] alg: No test for adler32 (adler32-zlib) [ 8205.694816] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8205.792611] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8207.384146] Key type lgssc registered [ 8207.778137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.586493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.592148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.733524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.099365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8216.000477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.129048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8217.576627] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8220.180816] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8222.159918] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8224.442900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8224.576701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.542520] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8228.942280] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8236.902898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8246.753192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.755188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8246.759094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.267020] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.267150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8247.274813] Lustre: Skipped 2 previous similar messages [ 8249.694280] Lustre: server umount lustre-MDT0000 complete [ 8250.923244] LustreError: 357659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145789 with bad export cookie 6852202298012740382 [ 8250.926707] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8250.927056] LustreError: 357659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8251.047631] Lustre: server umount lustre-MDT0001 complete [ 8262.512967] Lustre: server umount lustre-OST0000 complete [ 8267.394453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.595094] LustreError: 361407:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.606416] LustreError: 361407:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8267.627995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8269.160495] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8272.192583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.321758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8273.815690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8276.306262] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8278.148503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8280.548073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.681732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.646818] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8285.120554] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8291.944229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8302.563160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.564718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.567475] Lustre: Skipped 3 previous similar messages [ 8302.569913] Lustre: Skipped 1 previous similar message [ 8307.682130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.685020] Lustre: Skipped 3 previous similar messages [ 8308.647204] Lustre: server umount lustre-MDT0000 complete [ 8310.085568] LustreError: 361388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145848 with bad export cookie 6852202298012741481 [ 8310.089390] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8310.090608] LustreError: 361388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8310.211871] Lustre: server umount lustre-MDT0001 complete [ 8320.980571] Lustre: server umount lustre-OST0000 complete [ 8326.390104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.580481] LustreError: 365337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.607360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.153890] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8331.385229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.199139] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8335.946776] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8337.970834] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8340.473792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.742642] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8345.167767] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8350.887560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8357.345151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8357.347075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.350481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8361.443484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.446823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8363.149587] Lustre: server umount lustre-OST0000 complete [ 8366.563384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8366.565383] Lustre: Skipped 3 previous similar messages [ 8370.585599] Lustre: server umount lustre-MDT0000 complete [ 8371.680970] LustreError: 365356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.686274] LustreError: 365356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8371.913976] LustreError: 365319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145910 with bad export cookie 6852202298012742489 [ 8371.917533] LustreError: 365319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8371.917578] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8372.026759] Lustre: server umount lustre-MDT0001 complete [ 8375.922023] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8377.105787] Key type lgssc unregistered [ 8377.240445] LNet: 368884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8377.243471] LNetError: 368884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8377.254085] LNet: Removed LNI 192.168.204.146@tcp [ 8377.618337] Key type .llcrypt unregistered [ 8377.620035] Key type ._llcrypt unregistered [ 8387.264373] Key type ._llcrypt registered [ 8387.265485] Key type .llcrypt registered [ 8387.301219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8387.632990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8387.678531] alg: No test for adler32 (adler32-zlib) [ 8388.566848] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8388.652788] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8390.232214] Key type lgssc registered [ 8390.591791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8393.378816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8393.387965] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.392407] LustreError: 370301:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8393.405330] LustreError: 370301:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8393.409959] LustreError: 370301:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8393.413836] LustreError: 370301:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8393.418021] LustreError: 370301:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8396.306865] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8396.310857] LustreError: 370697:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8396.320242] LustreError: 370697:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8396.323364] LustreError: 370697:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8396.326172] LustreError: 370697:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8396.329618] LustreError: 370697:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8403.565459] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8404.635677] Key type lgssc unregistered [ 8404.773444] LNet: 371698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8404.776200] LNetError: 371698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8404.784275] LNet: Removed LNI 192.168.204.146@tcp [ 8405.080402] Key type .llcrypt unregistered [ 8405.081807] Key type ._llcrypt unregistered [ 8411.230733] Key type ._llcrypt registered [ 8411.231645] Key type .llcrypt registered [ 8411.266532] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8417.643027] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8418.053256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8418.088440] alg: No test for adler32 (adler32-zlib) [ 8418.963558] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8419.059852] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8420.656242] Key type lgssc registered [ 8421.086562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8433.138956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8433.147744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.232395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8434.247334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8434.285609] Lustre: lustre-MDT0000: new disk, initializing [ 8434.308445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.316144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8435.644458] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8439.835822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.882338] Lustre: 374827:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.908113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8439.910240] Lustre: Skipped 1 previous similar message [ 8439.944515] Lustre: lustre-MDT0001: new disk, initializing [ 8439.975865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8439.986902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8439.990475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8441.302086] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8445.053034] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8446.864266] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8449.257075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.363343] Lustre: lustre-OST0000: new disk, initializing [ 8449.365930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8449.369449] Lustre: 376797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8449.407132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.545093] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8455.502710] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8456.988518] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8458.241558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8458.245849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8458.282166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8459.126539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 8460.542731] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8460.626369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8463.330750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8463.331352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.333700] LustreError: Skipped 1 previous similar message [ 8463.337512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8463.340317] Lustre: Skipped 1 previous similar message [ 8467.410301] Lustre: server umount lustre-OST0000 complete [ 8471.008898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8471.011161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8471.015238] Lustre: Skipped 1 previous similar message [ 8471.017087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8473.569475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8473.569976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8478.689297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8478.691581] Lustre: Skipped 2 previous similar messages [ 8483.296138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8483.359186] Lustre: server umount lustre-MDT0000 complete [ 8484.644876] LustreError: 374819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146022 with bad export cookie 9848147851148427534 [ 8484.647112] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8484.650303] LustreError: 374819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8484.771401] Lustre: server umount lustre-MDT0001 complete [ 8487.756342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8488.451286] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:20:27 (1783146027) [ 8490.856782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.002340] LustreError: 379268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8491.010634] LustreError: 379268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8491.027751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.434719] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8495.422489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8495.552896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8496.973375] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8499.317678] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8501.150143] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8503.485269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8503.617708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8505.494711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8508.025472] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8510.360548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8510.424311] Lustre: lustre-OST0001: new disk, initializing [ 8510.427633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8510.432956] Lustre: 382100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8512.420260] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8515.583454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8515.587898] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8515.613406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8515.990134] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8520.673436] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8520.673632] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.675656] LustreError: Skipped 1 previous similar message [ 8520.679197] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8520.681433] Lustre: Skipped 1 previous similar message [ 8523.902763] Lustre: server umount lustre-OST0001 complete [ 8526.820802] LustreError: 382775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.826575] LustreError: 382775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8531.941186] LustreError: 382775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.948116] LustreError: 382775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8536.545810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.548293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.552425] Lustre: Skipped 1 previous similar message [ 8536.554240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.556630] Lustre: Skipped 3 previous similar messages [ 8537.062442] LustreError: 381088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.069195] LustreError: 381088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8542.182578] LustreError: 381086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.188907] LustreError: 381086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8550.369028] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8550.429901] Lustre: server umount lustre-OST0000 complete [ 8551.905395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8551.907598] LustreError: Skipped 1 previous similar message [ 8551.908921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.914480] Lustre: Skipped 1 previous similar message [ 8557.537554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8557.540075] Lustre: Skipped 10 previous similar messages [ 8566.240182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8566.304512] Lustre: server umount lustre-MDT0000 complete [ 8567.677177] LustreError: 379248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146105 with bad export cookie 9848147851148428402 [ 8567.680208] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.682708] LustreError: 379248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8573.875768] Lustre: server umount lustre-MDT0001 complete [ 8577.543990] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8578.577293] Key type lgssc unregistered [ 8578.717399] LNet: 384089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8578.720235] LNetError: 384089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8578.726327] LNet: Removed LNI 192.168.204.146@tcp [ 8579.058907] Key type .llcrypt unregistered [ 8579.060318] Key type ._llcrypt unregistered [ 8597.703602] Key type ._llcrypt registered [ 8597.704669] Key type .llcrypt registered [ 8597.739228] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8598.146427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8598.154726] alg: No test for adler32 (adler32-zlib) [ 8599.005082] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8599.094041] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8600.672165] Key type lgssc registered [ 8601.027604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8603.552909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8603.559813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.642110] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8604.649370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8604.700548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8606.056783] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8608.946408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.981406] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8608.991152] Lustre: 386863:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8609.084799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.434632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8612.799912] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8614.534372] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8616.929409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8616.929996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8616.933570] Lustre: Skipped 1 previous similar message [ 8621.332686] Lustre: server umount lustre-MDT0000 complete [ 8622.603597] LustreError: 386855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146160 with bad export cookie 17430234614544232463 [ 8622.606050] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8622.607952] LustreError: 386855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8622.714229] Lustre: server umount lustre-MDT0001 complete [ 8625.833074] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:22:44 (1783146164) [ 8627.955401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.103668] LustreError: 389372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.110389] LustreError: 389372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8628.120564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.377088] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8632.079725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.193209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8633.569093] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8635.989333] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8637.775983] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8639.987665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8640.053974] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8640.116466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.020429] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8644.380515] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8653.304258] Lustre: Permanently deactivating lustre-OST0000 [ 8653.309316] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8661.489793] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8662.921824] LustreError: 392502:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8662.925451] LustreError: 392502:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8662.927696] LustreError: 392502:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8663.537423] LustreError: 392550:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8663.540974] LustreError: 392550:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8670.286349] Lustre: server umount lustre-OST0000 complete [ 8671.167754] LustreError: 392800:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8671.169846] LustreError: 392800:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8673.248820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8673.251281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.255540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8673.257494] Lustre: Skipped 1 previous similar message [ 8675.297103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8675.297693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8680.416873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8680.418669] Lustre: Skipped 2 previous similar messages [ 8685.536929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.539045] Lustre: Skipped 1 previous similar message [ 8686.048096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8686.101662] Lustre: server umount lustre-MDT0000 complete [ 8687.301274] LustreError: 392063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146225 with bad export cookie 17430234614544233030 [ 8687.304666] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8687.307748] LustreError: 392063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8697.877875] Lustre: server umount lustre-MDT0001 complete [ 8700.817311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.214607] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8703.121952] LustreError: 393959:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8705.134630] Lustre: 394297:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [ 8705.825100] Lustre: server umount MGS complete [ 8709.473296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.608705] LustreError: 395020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.613652] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8709.618101] Lustre: Skipped 1 previous similar message [ 8709.637228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8710.962926] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8713.750365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.847173] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8715.124706] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8717.460680] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8719.246431] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8719.889177] Lustre: Permanently reactivating lustre-OST0000 [ 8719.894701] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8719.896420] Lustre: Skipped 2 previous similar messages [ 8744.928691] LustreError: 396797:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8748.465934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.586441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8748.589655] Lustre: Skipped 1 previous similar message [ 8750.582485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8753.015307] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8764.896910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8764.900179] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8764.904568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8765.923309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8770.843976] Lustre: server umount lustre-OST0000 complete [ 8775.648888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8775.652319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8775.657276] Lustre: Skipped 1 previous similar message [ 8778.465974] Lustre: server umount lustre-MDT0000 complete [ 8779.751178] LustreError: 395000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146317 with bad export cookie 17430234614544234066 [ 8779.753769] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8779.755361] LustreError: 395000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8779.873442] Lustre: server umount lustre-MDT0001 complete [ 8783.718679] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8784.775158] Key type lgssc unregistered [ 8784.904350] LNet: 399691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8784.907578] LNetError: 399691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8784.916321] LNet: Removed LNI 192.168.204.146@tcp [ 8785.243053] Key type .llcrypt unregistered [ 8785.244026] Key type ._llcrypt unregistered [ 8791.235626] Key type ._llcrypt registered [ 8791.236760] Key type .llcrypt registered [ 8791.272832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8797.833683] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8798.240166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8798.275641] alg: No test for adler32 (adler32-zlib) [ 8799.130597] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8799.219838] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8800.800126] Key type lgssc registered [ 8801.158411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8812.852589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8812.858455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.957085] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8813.969914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8814.007843] Lustre: lustre-MDT0000: new disk, initializing [ 8814.029285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.035717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8815.435841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8819.733838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.782107] Lustre: 402819:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8819.812422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8819.814375] Lustre: Skipped 1 previous similar message [ 8819.849848] Lustre: lustre-MDT0001: new disk, initializing [ 8819.884513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8819.897078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8819.900718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8821.279859] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8824.988963] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8826.869676] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8829.380162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.484357] Lustre: lustre-OST0000: new disk, initializing [ 8829.486585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8829.489757] Lustre: 404786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8829.522488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.594591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8835.317370] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8836.602735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8836.606502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8836.643071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8836.676224] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8836.744207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8838.096139] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8838.164477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8841.698034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8841.698901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.699529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8841.700242] LustreError: Skipped 1 previous similar message [ 8841.706382] Lustre: Skipped 1 previous similar message [ 8844.889292] Lustre: server umount lustre-OST0000 complete [ 8846.817251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8846.817908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8846.821042] Lustre: Skipped 1 previous similar message [ 8846.823372] Lustre: Skipped 1 previous similar message [ 8851.937027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8851.939075] Lustre: Skipped 2 previous similar messages [ 8852.311413] Lustre: server umount lustre-MDT0000 complete [ 8853.519395] LustreError: 402857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146391 with bad export cookie 17665520690521972646 [ 8853.521383] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8853.523131] LustreError: 402857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8853.643857] Lustre: server umount lustre-MDT0001 complete [ 8856.388874] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8856.960324] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:26:35 (1783146395) [ 8859.240312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8859.416835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8860.786500] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8863.687475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.813361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8865.296511] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8867.704680] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8869.573254] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8871.891531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8872.000987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8874.043518] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8876.588503] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8877.275974] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8887.950289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8890.851145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8890.852012] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8890.852200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8890.856431] Lustre: Skipped 1 previous similar message [ 8895.506555] Lustre: server umount lustre-OST0000 complete [ 8899.552736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8899.555373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8899.559846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8899.562196] Lustre: Skipped 1 previous similar message [ 8911.328128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8911.329215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8911.337795] Lustre: Skipped 5 previous similar messages [ 8911.394152] Lustre: server umount lustre-MDT0000 complete [ 8912.655029] LustreError: 407224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146450 with bad export cookie 17665520690521973514 [ 8912.658081] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8912.658644] LustreError: 407224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8912.776114] Lustre: server umount lustre-MDT0001 complete [ 8916.601658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8917.705743] Key type lgssc unregistered [ 8917.844403] LNet: 410987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8917.847361] LNetError: 410987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8917.855326] LNet: Removed LNI 192.168.204.146@tcp [ 8918.189122] Key type .llcrypt unregistered [ 8918.190658] Key type ._llcrypt unregistered [ 8926.030033] Key type ._llcrypt registered [ 8926.031663] Key type .llcrypt registered [ 8926.071717] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8926.182735] Key type .llcrypt unregistered [ 8926.184045] Key type ._llcrypt unregistered [ 8933.326204] Key type ._llcrypt registered [ 8933.327469] Key type .llcrypt registered [ 8933.365927] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8933.761447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8933.776875] alg: No test for adler32 (adler32-zlib) [ 8934.628611] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8934.718601] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8936.296177] Key type lgssc registered [ 8936.706670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8939.261887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8939.267716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.395762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8941.698342] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8944.414045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8944.523330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.430553] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8948.769104] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8949.730505] LustreError: 413185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.735688] LustreError: 413185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8950.754711] LustreError: 413184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.759518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8950.761071] LustreError: 413184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8951.001106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.134169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8952.467335] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8960.992825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.995971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8961.001116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8961.508114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8961.508955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8961.514697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8965.704906] Lustre: server umount lustre-OST0000 complete [ 8970.722047] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8970.724650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.729044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8970.731039] Lustre: Skipped 1 previous similar message [ 8972.946991] Lustre: server umount lustre-MDT0000 complete [ 8974.067228] LustreError: 413165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146512 with bad export cookie 2678102532268254991 [ 8974.069587] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8974.072171] LustreError: 413165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8974.178690] Lustre: server umount lustre-MDT0001 complete [ 8977.358070] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8978.290642] Key type lgssc unregistered [ 8978.419441] LNet: 416330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8978.422791] LNetError: 416330:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8978.430249] LNet: Removed LNI 192.168.204.146@tcp [ 8978.711148] Key type .llcrypt unregistered [ 8978.712147] Key type ._llcrypt unregistered [ 8987.097557] Key type ._llcrypt registered [ 8987.099013] Key type .llcrypt registered [ 8987.134394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8987.431219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.437594] alg: No test for adler32 (adler32-zlib) [ 8988.293323] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 8988.394646] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8989.976178] Key type lgssc registered [ 8990.346880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8992.885236] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8992.891949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.352369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9013.472316] LustreError: 417697:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 9040.096179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9041.977118] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9044.313912] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9046.461775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.643560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9046.666088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9047.910119] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9050.532079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.650244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9051.994848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9054.283447] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9056.015994] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9060.833349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9060.835623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.839176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9061.858957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9061.859139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9064.274046] Lustre: server umount lustre-OST0000 complete [ 9065.952691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9065.954935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.958331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9065.959760] Lustre: Skipped 1 previous similar message [ 9071.637122] Lustre: server umount lustre-MDT0000 complete [ 9072.096961] LustreError: 418737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.102352] LustreError: 418737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9072.854505] LustreError: 417704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146611 with bad export cookie 17031158021743137355 [ 9072.856907] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9072.859847] LustreError: 417704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9072.972190] Lustre: server umount lustre-MDT0001 complete [ 9076.327328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9077.253095] Key type lgssc unregistered [ 9077.363332] LNet: 421263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9077.365864] LNetError: 421263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9077.372241] LNet: Removed LNI 192.168.204.146@tcp [ 9077.666509] Key type .llcrypt unregistered [ 9077.667965] Key type ._llcrypt unregistered [ 9086.162167] Key type ._llcrypt registered [ 9086.163500] Key type .llcrypt registered [ 9086.203617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9086.543324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9086.581991] alg: No test for adler32 (adler32-zlib) [ 9087.435283] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 9087.525214] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9089.104139] Key type lgssc registered [ 9089.429706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9091.879855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9091.886274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.014805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9094.270711] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9097.053450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9097.195955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9098.601876] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9100.921338] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9102.639297] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9104.819495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9104.925098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.798979] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9109.020787] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9118.246896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9122.785052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9122.788679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9122.794252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.297571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.298533] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.300281] Lustre: Skipped 1 previous similar message [ 9123.303280] Lustre: Skipped 1 previous similar message [ 9125.459209] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [ 9127.317023] Lustre: server umount lustre-MDT0000 complete [ 9128.125579] LustreError: 422636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.46@tcp arrived at 1783146666 with bad export cookie 7611304919622183312 [ 9128.131186] LustreError: 422636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9128.418158] LustreError: 422654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.424917] LustreError: 422654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9128.809978] LustreError: 422637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146666 with bad export cookie 7611304919622182661 [ 9128.814133] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9128.815290] LustreError: 422637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9132.513283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9132.515779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.521117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9133.537317] LustreError: 423396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9133.545542] LustreError: 423396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9134.929165] Lustre: server umount lustre-OST0000 complete [ 9137.284621] Lustre: server umount lustre-MDT0001 complete [ 9140.722808] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9141.701048] Key type lgssc unregistered [ 9141.835350] LNet: 426358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9141.838334] LNetError: 426358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9141.844411] LNet: Removed LNI 192.168.204.146@tcp [ 9142.172948] Key type .llcrypt unregistered [ 9142.174301] Key type ._llcrypt unregistered [ 9150.963238] Key type ._llcrypt registered [ 9150.964284] Key type .llcrypt registered [ 9150.999116] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9151.276165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9151.285148] alg: No test for adler32 (adler32-zlib) [ 9152.132054] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 9152.217901] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9153.792119] Key type lgssc registered [ 9154.116141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9156.533751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9156.539655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.667671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.947735] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9161.566371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9161.683215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.922705] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9165.207785] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9166.972193] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9169.182243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.275961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9171.038124] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9173.264447] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9182.629570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9187.297808] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9187.300714] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.305857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9187.810946] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9187.816887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9188.946786] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 9190.806915] Lustre: server umount lustre-MDT0001 complete [ 9192.929282] LustreError: 427768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.929876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.934034] LustreError: 427768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9192.938424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9192.938720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.049208] LustreError: 427750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9198.049479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.475221] Lustre: server umount lustre-OST0000 complete [ 9199.842545] Lustre: server umount lustre-MDT0000 complete [ 9204.218888] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9205.213577] Key type lgssc unregistered [ 9205.351366] LNet: 431455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9205.353664] LNetError: 431455:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9205.362385] LNet: Removed LNI 192.168.204.146@tcp [ 9205.703885] Key type .llcrypt unregistered [ 9205.705130] Key type ._llcrypt unregistered [ 9213.661595] Key type ._llcrypt registered [ 9213.662801] Key type .llcrypt registered [ 9213.699485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9219.510865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9219.810419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9219.856038] alg: No test for adler32 (adler32-zlib) [ 9220.701132] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 9220.782194] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9222.352148] Key type lgssc registered [ 9222.683403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9234.179766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9239.087460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9239.093501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9240.167672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9240.177868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9240.210222] Lustre: lustre-MDT0000: new disk, initializing [ 9240.227840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9240.233582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9241.584569] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9247.088037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9247.178941] Lustre: lustre-OST0000: new disk, initializing [ 9247.181599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9247.184094] Lustre: Skipped 1 previous similar message [ 9247.186783] Lustre: 436458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9247.210329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.167047] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9252.703234] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9254.908835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9254.912504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9254.922660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9262.974756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.019536] Lustre: 435441:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9263.043857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9263.078300] Lustre: lustre-MDT0001: new disk, initializing [ 9263.117882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.131147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 9263.134503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 9264.345413] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9267.798709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9267.858337] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9269.104272] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9269.169065] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9272.722323] Lustre: server umount lustre-MDT0001 complete [ 9275.364491] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9275.364845] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9280.482810] LustreError: 436198:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.487244] LustreError: 436198:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9285.601593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9285.601813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.602107] LustreError: 437783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.611087] Lustre: Skipped 1 previous similar message [ 9285.612747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.722339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.722620] LustreError: 435452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.031111] Lustre: server umount lustre-OST0000 complete [ 9298.492403] Lustre: server umount lustre-MDT0000 complete [ 9303.118547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9304.171599] Key type lgssc unregistered [ 9304.300440] LNet: 440204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9304.304487] LNetError: 440204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9304.313353] LNet: Removed LNI 192.168.204.146@tcp [ 9304.612536] Key type .llcrypt unregistered [ 9304.613622] Key type ._llcrypt unregistered [ 9316.546784] Key type ._llcrypt registered [ 9316.547761] Key type .llcrypt registered [ 9316.580891] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9316.664536] Key type .llcrypt unregistered [ 9316.665500] Key type ._llcrypt unregistered [ 9322.640984] Key type ._llcrypt registered [ 9322.642305] Key type .llcrypt registered [ 9322.676132] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9322.962667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9322.992107] alg: No test for adler32 (adler32-zlib) [ 9323.834585] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [ 9323.909749] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9325.480114] Key type lgssc registered [ 9325.788237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9334.232887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9334.237724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.321856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9335.333591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9335.370764] Lustre: lustre-MDT0000: new disk, initializing [ 9335.393485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9335.400598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9336.767253] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9340.892973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9340.936824] Lustre: 443550:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9340.961243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9340.963383] Lustre: Skipped 1 previous similar message [ 9340.997763] Lustre: lustre-MDT0001: new disk, initializing [ 9341.029703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9341.042281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9341.045630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9342.433372] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9346.106588] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9347.888718] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9350.154488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.260298] Lustre: lustre-OST0000: new disk, initializing [ 9350.262147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9350.265091] Lustre: 445519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9350.295822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.365045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9351.370254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9351.408039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9352.244112] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9355.801615] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9361.376904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9361.379565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9361.384703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9361.890644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9361.891331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9361.897294] Lustre: Skipped 1 previous similar message [ 9361.900483] Lustre: Skipped 1 previous similar message [ 9364.184442] Lustre: server umount lustre-MDT0000 complete [ 9365.364305] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146903 with bad export cookie 1377175894255541444 [ 9365.367445] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9365.367870] LustreError: 444346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9365.481067] Lustre: server umount lustre-MDT0001 complete [ 9376.931492] Lustre: server umount lustre-OST0000 complete [ 9380.253436] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:35:18 (1783146918) [ 9382.761573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.940732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.260440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9386.985378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9387.099587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9388.408090] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9390.762591] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9392.588429] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9407.907224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9407.973592] Lustre: Found index 0 for lustre-OST0000, updating log [ 9408.032195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.066874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9409.945817] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9412.278837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9414.112782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.115352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.119421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9417.187336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9417.188067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.188708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.469982] Lustre: server umount lustre-OST0000 complete [ 9431.976744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9431.977440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9431.980108] Lustre: Skipped 1 previous similar message [ 9431.983042] Lustre: Skipped 2 previous similar messages [ 9444.321461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9444.323177] Lustre: Skipped 7 previous similar messages [ 9446.368192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9446.431368] Lustre: server umount lustre-MDT0000 complete [ 9447.675781] LustreError: 447524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146985 with bad export cookie 1377175894255542823 [ 9447.680196] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9447.803666] Lustre: server umount lustre-MDT0001 complete [ 9450.581838] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:36:29 (1783146989) [ 9472.416194] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [10030.108173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.251187] LustreError: 453943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.259735] LustreError: 453943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10030.274946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10031.610568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10034.353340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.797392] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10038.143667] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10039.860172] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10055.143643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.233803] Lustre: Found index 0 for lustre-OST0000, updating log [10057.346103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10057.384547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [10059.726120] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10061.411685] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10061.416404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10066.511948] Lustre: server umount lustre-OST0000 complete [10067.938168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.941855] Lustre: Skipped 1 previous similar message [10082.272140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10082.340956] Lustre: server umount lustre-MDT0000 complete [10083.298076] LustreError: 453943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.309432] LustreError: 453943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10083.634153] LustreError: 455844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147621 with bad export cookie 1377175894255543894 [10083.637336] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10083.637928] LustreError: 455844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10083.757085] Lustre: server umount lustre-MDT0001 complete [10085.792173] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [10101.282709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10104.454545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10115.948747] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:47:34 (1783147654) [10121.554895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10122.572179] Key type lgssc unregistered [10122.699420] LNet: 460596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10122.701826] LNetError: 460596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10184.166297] LNet: Removed LNI 192.168.204.146@tcp [10184.476584] Key type .llcrypt unregistered [10184.477820] Key type ._llcrypt unregistered [10191.170380] Key type ._llcrypt registered [10191.171841] Key type .llcrypt registered [10191.209945] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10191.524469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10191.535493] alg: No test for adler32 (adler32-zlib) [10192.405669] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10192.498143] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10194.080125] Key type lgssc registered [10194.430699] Lustre: Echo OBD driver; http://www.lustre.org/ [10195.854643] LNet: Added LNI 192.168.204.146@tcp100 [8/256/0/180] [10196.146569] LNet: Added LNI 192.168.204.146@tcp101 [8/256/0/180] [10196.443696] LNet: Added LNI 192.168.204.146@tcp102 [8/256/0/180] [10196.735364] LNet: Added LNI 192.168.204.146@tcp103 [8/256/0/180] [10197.026952] LNet: Added LNI 192.168.204.146@tcp104 [8/256/0/180] [10197.310179] LNet: Added LNI 192.168.204.146@tcp105 [8/256/0/180] [10197.601875] LNet: Added LNI 192.168.204.146@tcp106 [8/256/0/180] [10197.894067] LNet: Added LNI 192.168.204.146@tcp107 [8/256/0/180] [10198.178238] LNet: Added LNI 192.168.204.146@tcp108 [8/256/0/180] [10198.459666] LNet: Added LNI 192.168.204.146@tcp109 [8/256/0/180] [10198.755604] LNet: Added LNI 192.168.204.146@tcp110 [8/256/0/180] [10199.046334] LNet: Added LNI 192.168.204.146@tcp111 [8/256/0/180] [10199.345722] LNet: Added LNI 192.168.204.146@tcp112 [8/256/0/180] [10199.639972] LNet: Added LNI 192.168.204.146@tcp113 [8/256/0/180] [10199.935507] LNet: Added LNI 192.168.204.146@tcp114 [8/256/0/180] [10200.233595] LNet: Added LNI 192.168.204.146@tcp115 [8/256/0/180] [10200.518297] LNet: Added LNI 192.168.204.146@tcp116 [8/256/0/180] [10200.806666] LNet: Added LNI 192.168.204.146@tcp117 [8/256/0/180] [10201.109132] LNet: Added LNI 192.168.204.146@tcp118 [8/256/0/180] [10201.404832] LNet: Added LNI 192.168.204.146@tcp119 [8/256/0/180] [10201.699223] LNet: Added LNI 192.168.204.146@tcp120 [8/256/0/180] [10201.999400] LNet: Added LNI 192.168.204.146@tcp121 [8/256/0/180] [10202.303901] LNet: Added LNI 192.168.204.146@tcp122 [8/256/0/180] [10202.580618] LNet: Added LNI 192.168.204.146@tcp123 [8/256/0/180] [10202.862483] LNet: Added LNI 192.168.204.146@tcp124 [8/256/0/180] [10203.144889] LNet: Added LNI 192.168.204.146@tcp125 [8/256/0/180] [10203.428114] LNet: Added LNI 192.168.204.146@tcp126 [8/256/0/180] [10203.705276] LNet: Added LNI 192.168.204.146@tcp127 [8/256/0/180] [10203.992494] LNet: Added LNI 192.168.204.146@tcp128 [8/256/0/180] [10204.285186] LNet: Added LNI 192.168.204.146@tcp129 [8/256/0/180] [10204.579527] LNet: Added LNI 192.168.204.146@tcp130 [8/256/0/180] [10204.869048] LNet: Added LNI 192.168.204.146@tcp131 [8/256/0/180] [10205.155188] LNet: Added LNI 192.168.204.146@tcp132 [8/256/0/180] [10205.438916] LNet: Added LNI 192.168.204.146@tcp133 [8/256/0/180] [10205.728013] LNet: Added LNI 192.168.204.146@tcp134 [8/256/0/180] [10206.015423] LNet: Added LNI 192.168.204.146@tcp135 [8/256/0/180] [10208.131220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10208.136621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10209.171028] Lustre: 463592:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [10209.203455] LNetError: 463613:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10209.217103] LNetError: Protocol error connecting to 192.168.204.146@tcp122 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10209.221336] LNetError: Skipped 1 previous similar message [10209.266191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10209.275876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10209.311435] Lustre: lustre-MDT0000: new disk, initializing [10209.332839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10209.338684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10209.888102] LNet: 4 local NIs in recovery (showing 4): 192.168.204.146@tcp126, 192.168.204.146@tcp127, 192.168.204.146@tcp122, 192.168.204.146@tcp129 [10210.274741] LNetError: 463616:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10210.277562] LNetError: 463616:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [10210.279810] LNetError: Protocol error connecting to 192.168.204.146@tcp113 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10210.282621] LNetError: Skipped 5 previous similar messages [10210.620830] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10212.964131] LNetError: 461055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10212.967422] LNetError: 461055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [10212.969981] LNetError: Protocol error connecting to 192.168.204.146@tcp122 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10212.973145] LNetError: Skipped 4 previous similar messages [10214.706712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.752214] Lustre: 463607:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10214.775501] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10214.777249] Lustre: Skipped 1 previous similar message [10214.810629] Lustre: lustre-MDT0001: new disk, initializing [10214.823069] Lustre: 464413:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [10214.827072] Lustre: 464413:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [10214.864099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10214.876081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10214.879936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10216.033292] LNetError: 463618:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10216.037787] LNetError: Protocol error connecting to 192.168.204.146@tcp122 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10216.131174] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10219.703191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10221.338038] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10223.563242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10223.680207] Lustre: lustre-OST0000: new disk, initializing [10223.681900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10223.684619] Lustre: 465585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10223.727954] Lustre: 465590:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.146@tcp: can't add NID 192.168.204.146@tcp130: rc = -75 [10223.730820] Lustre: 465590:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [10223.761491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10223.811210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10223.815967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10223.825558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [10224.225574] LNetError: 463613:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10224.228908] LNetError: Protocol error connecting to 192.168.204.146@tcp122 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10225.664855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10229.383760] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10234.341632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10234.342328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10234.344451] LustreError: Skipped 1 previous similar message [10234.348368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10234.351759] Lustre: Skipped 1 previous similar message [10237.710792] Lustre: server umount lustre-OST0000 complete [10239.458201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.460626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.464692] Lustre: Skipped 2 previous similar messages [10239.585642] LNetError: 463620:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10239.591831] LNetError: Protocol error connecting to 192.168.204.146@tcp111 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10244.577711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10244.580958] Lustre: Skipped 2 previous similar messages [10249.697120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10249.699833] Lustre: Skipped 1 previous similar message [10253.280158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10253.373037] Lustre: server umount lustre-MDT0000 complete [10254.632240] LustreError: 463602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147792 with bad export cookie 67985576470781431 [10254.635665] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10254.637565] LustreError: 463602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10254.759941] Lustre: server umount lustre-MDT0001 complete [10256.993416] LNetError: 461055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.146 [10256.996640] LNetError: 461055:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [10256.999169] LNetError: Protocol error connecting to 192.168.204.146@tcp125 at host 192.168.204.146:988: is it running a compatible version of Lustre? [10257.002638] LNetError: Skipped 7 previous similar messages [10258.422602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10259.422348] Key type lgssc unregistered [10259.571912] LNet: 467499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10259.576644] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10259.580794] LNet: Removed LNI 192.168.204.146@tcp [10260.640149] LNet: Removed LNI 192.168.204.146@tcp100 [10260.641672] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10260.644701] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10260.648684] LNet: Removed LNI 192.168.204.146@tcp101 [10261.664134] LNet: Removed LNI 192.168.204.146@tcp102 [10261.666196] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10261.669640] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10261.673591] LNet: Removed LNI 192.168.204.146@tcp103 [10262.688166] LNet: Removed LNI 192.168.204.146@tcp104 [10263.712137] LNet: Removed LNI 192.168.204.146@tcp105 [10263.713647] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10263.715993] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10263.718586] LNet: Removed LNI 192.168.204.146@tcp106 [10264.736202] LNet: Removed LNI 192.168.204.146@tcp107 [10265.760193] LNet: Removed LNI 192.168.204.146@tcp108 [10266.784158] LNet: Removed LNI 192.168.204.146@tcp109 [10267.808128] LNet: Removed LNI 192.168.204.146@tcp110 [10267.809598] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10267.811952] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10267.814390] LNet: Removed LNI 192.168.204.146@tcp111 [10268.832156] LNet: Removed LNI 192.168.204.146@tcp112 [10269.856126] LNet: Removed LNI 192.168.204.146@tcp113 [10270.880163] LNet: Removed LNI 192.168.204.146@tcp114 [10271.904157] LNet: Removed LNI 192.168.204.146@tcp115 [10272.928135] LNet: Removed LNI 192.168.204.146@tcp116 [10273.952105] LNet: Removed LNI 192.168.204.146@tcp117 [10274.976144] LNet: Removed LNI 192.168.204.146@tcp118 [10276.000101] LNet: Removed LNI 192.168.204.146@tcp119 [10276.001497] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10276.004327] LNetError: 467499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10276.006460] LNet: Removed LNI 192.168.204.146@tcp120 [10277.024105] LNet: Removed LNI 192.168.204.146@tcp121 [10278.048112] LNet: Removed LNI 192.168.204.146@tcp122 [10279.072108] LNet: Removed LNI 192.168.204.146@tcp123 [10280.096178] LNet: Removed LNI 192.168.204.146@tcp124 [10281.120140] LNet: Removed LNI 192.168.204.146@tcp125 [10282.144127] LNet: Removed LNI 192.168.204.146@tcp126 [10283.168171] LNet: Removed LNI 192.168.204.146@tcp127 [10284.192160] LNet: Removed LNI 192.168.204.146@tcp128 [10285.216129] LNet: Removed LNI 192.168.204.146@tcp129 [10285.217692] LNet: Removed LNI 192.168.204.146@tcp130 [10285.218918] LNet: Removed LNI 192.168.204.146@tcp131 [10285.220193] LNet: Removed LNI 192.168.204.146@tcp132 [10285.221395] LNet: Removed LNI 192.168.204.146@tcp133 [10285.222539] LNet: Removed LNI 192.168.204.146@tcp134 [10285.229541] LNet: Removed LNI 192.168.204.146@tcp135 [10285.515605] Key type .llcrypt unregistered [10285.518631] Key type ._llcrypt unregistered [10292.549736] Key type ._llcrypt registered [10292.550811] Key type .llcrypt registered [10292.584495] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10298.355647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10298.696256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10298.750621] alg: No test for adler32 (adler32-zlib) [10299.605960] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10299.690895] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10301.272166] Key type lgssc registered [10301.661829] Lustre: Echo OBD driver; http://www.lustre.org/ [10312.352351] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:50:51 (1783147851) [10317.895250] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10318.922650] Key type lgssc unregistered [10319.044280] LNet: 471557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10319.046984] LNetError: 471557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10319.055257] LNet: Removed LNI 192.168.204.146@tcp [10319.335398] Key type .llcrypt unregistered [10319.336551] Key type ._llcrypt unregistered [10325.727293] Key type ._llcrypt registered [10325.728575] Key type .llcrypt registered [10325.765383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10326.123989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10326.144759] alg: No test for adler32 (adler32-zlib) [10327.006180] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10327.094892] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10328.680185] Key type lgssc registered [10329.105349] Lustre: Echo OBD driver; http://www.lustre.org/ [10332.182807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10332.187978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.271218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10333.280973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10333.315212] Lustre: lustre-MDT0000: new disk, initializing [10333.334603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10333.340317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10334.686756] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10339.064451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.110650] Lustre: 472745:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10339.136249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10339.139152] Lustre: Skipped 1 previous similar message [10339.177857] Lustre: lustre-MDT0001: new disk, initializing [10339.214563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10339.226915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10339.230042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10340.583758] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10344.335488] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10346.214848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10348.532776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10348.660785] Lustre: lustre-OST0000: new disk, initializing [10348.662720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10348.666142] Lustre: 474718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10348.704605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10350.732377] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10354.395516] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10355.648352] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing lnet_if_list [10356.238442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10356.245075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10356.289977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10357.323671] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10357.335665] LNet: Added LNI 192.168.204.146@tcp10 [8/256/0/180] [10359.066698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10359.078224] LNet: Added LNI 192.168.204.146@tcp11 [8/256/0/180] [10360.375113] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10360.384835] LNet: Added LNI 192.168.204.146@tcp12 [8/256/0/180] [10361.706524] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10361.716651] LNet: Added LNI 192.168.204.146@tcp13 [8/256/0/180] [10363.060306] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10363.073099] LNet: Added LNI 192.168.204.146@tcp14 [8/256/0/180] [10364.506316] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10364.517371] LNet: Added LNI 192.168.204.146@tcp15 [8/256/0/180] [10365.947441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10365.959808] LNet: Added LNI 192.168.204.146@tcp16 [8/256/0/180] [10367.442796] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10367.454661] LNet: Added LNI 192.168.204.146@tcp17 [8/256/0/180] [10368.959388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10368.969809] LNet: Added LNI 192.168.204.146@tcp18 [8/256/0/180] [10370.589196] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10370.600162] LNet: Added LNI 192.168.204.146@tcp19 [8/256/0/180] [10372.190096] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10372.200563] LNet: Added LNI 192.168.204.146@tcp20 [8/256/0/180] [10373.905709] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10373.917196] LNet: Added LNI 192.168.204.146@tcp21 [8/256/0/180] [10375.663818] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10375.674033] LNet: Added LNI 192.168.204.146@tcp22 [8/256/0/180] [10377.339383] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10377.352534] LNet: Added LNI 192.168.204.146@tcp23 [8/256/0/180] [10379.035650] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10379.046814] LNet: Added LNI 192.168.204.146@tcp24 [8/256/0/180] [10380.843413] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10380.853950] LNet: Added LNI 192.168.204.146@tcp25 [8/256/0/180] [10382.597542] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10382.608918] LNet: Added LNI 192.168.204.146@tcp26 [8/256/0/180] [10384.579867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10384.591155] LNet: Added LNI 192.168.204.146@tcp27 [8/256/0/180] [10386.537933] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10386.549272] LNet: Added LNI 192.168.204.146@tcp28 [8/256/0/180] [10388.412516] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10388.427872] LNet: Added LNI 192.168.204.146@tcp29 [8/256/0/180] [10390.249812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10390.260507] LNet: Added LNI 192.168.204.146@tcp30 [8/256/0/180] [10392.162297] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10392.175518] LNet: Added LNI 192.168.204.146@tcp31 [8/256/0/180] [10394.127395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10394.137163] LNet: Added LNI 192.168.204.146@tcp32 [8/256/0/180] [10396.168471] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10396.178728] LNet: Added LNI 192.168.204.146@tcp33 [8/256/0/180] [10398.186156] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10398.195734] LNet: Added LNI 192.168.204.146@tcp34 [8/256/0/180] [10400.239293] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10400.251251] LNet: Added LNI 192.168.204.146@tcp35 [8/256/0/180] [10402.406849] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10402.418536] LNet: Added LNI 192.168.204.146@tcp36 [8/256/0/180] [10404.861985] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10404.873333] LNet: Added LNI 192.168.204.146@tcp37 [8/256/0/180] [10407.033923] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10407.045990] LNet: Added LNI 192.168.204.146@tcp38 [8/256/0/180] [10409.220789] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10409.234294] LNet: Added LNI 192.168.204.146@tcp39 [8/256/0/180] [10411.438869] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10411.449826] LNet: Added LNI 192.168.204.146@tcp40 [8/256/0/180] [10413.740510] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10413.752727] LNet: Added LNI 192.168.204.146@tcp41 [8/256/0/180] [10416.082824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10416.093292] LNet: Added LNI 192.168.204.146@tcp42 [8/256/0/180] [10418.491348] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10418.503964] LNet: Added LNI 192.168.204.146@tcp43 [8/256/0/180] [10420.911318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10420.927936] LNet: Added LNI 192.168.204.146@tcp44 [8/256/0/180] [10423.422433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10423.435341] LNet: Added LNI 192.168.204.146@tcp45 [8/256/0/180] [10426.127674] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10426.139916] LNet: Added LNI 192.168.204.146@tcp46 [8/256/0/180] [10428.765300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10428.775839] LNet: Added LNI 192.168.204.146@tcp47 [8/256/0/180] [10431.282465] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10431.294903] LNet: Added LNI 192.168.204.146@tcp48 [8/256/0/180] [10433.721585] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10433.736425] LNet: Added LNI 192.168.204.146@tcp49 [8/256/0/180] [10436.284089] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10436.294254] LNet: Added LNI 192.168.204.146@tcp50 [8/256/0/180] [10446.816959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10446.819354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.823231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10448.354719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10448.355128] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10448.356922] Lustre: Skipped 1 previous similar message [10448.363377] Lustre: Skipped 1 previous similar message [10449.628759] Lustre: server umount lustre-MDT0000 complete [10450.778148] LustreError: 473543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147988 with bad export cookie 8304265006295438901 [10450.781742] LustreError: 473543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10450.781828] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10450.884939] Lustre: server umount lustre-MDT0001 complete [10458.013191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10460.895956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.856095] Lustre: 472024:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147991/real 1783147991] req@000000009697f6b4 x1869766055771392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783148007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10469.863746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10470.880141] Lustre: 472024:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147993/real 1783147993] req@0000000016b36dec x1869766055771520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783148009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10473.952117] Lustre: 472021:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147996/real 1783147996] req@000000007049667e x1869766055771648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783148012 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10474.976458] LustreError: 472020:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005795f7aa x1869766055772544/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10475.090693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10476.129107] Lustre: 472021:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147998/real 1783147998] req@00000000af5b4d80 x1869766055771904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783148014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10476.413232] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10479.375715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.534004] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10480.549943] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10481.126368] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10481.568083] Lustre: 472023:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783148003/real 1783148003] req@0000000097a792a9 x1869766055772032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783148019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10483.521327] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10485.383795] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10488.744708] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10493.922970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.928499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10493.931511] Lustre: Skipped 2 previous similar messages [10498.452602] Lustre: server umount lustre-OST0000 complete [10500.064923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10500.067520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.071902] Lustre: Skipped 1 previous similar message [10500.074415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.162497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.168117] Lustre: Skipped 1 previous similar message [10505.830976] Lustre: server umount lustre-MDT0000 complete [10507.294340] LustreError: 473543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148045 with bad export cookie 8304265006295440938 [10507.298507] LustreError: 473543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10507.298736] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10507.428907] Lustre: server umount lustre-MDT0001 complete [10511.681968] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10512.866806] Key type lgssc unregistered [10513.026589] LNet: 501674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10513.032679] LNetError: 501674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10513.037536] LNet: Removed LNI 192.168.204.146@tcp [10513.040171] LNet: Removed LNI 192.168.204.146@tcp10 [10513.042275] LNet: Removed LNI 192.168.204.146@tcp11 [10513.044468] LNet: Removed LNI 192.168.204.146@tcp12 [10513.046546] LNet: Removed LNI 192.168.204.146@tcp13 [10513.048587] LNet: Removed LNI 192.168.204.146@tcp14 [10513.050519] LNet: Removed LNI 192.168.204.146@tcp15 [10513.052427] LNet: Removed LNI 192.168.204.146@tcp16 [10513.054096] LNet: Removed LNI 192.168.204.146@tcp17 [10513.055708] LNet: Removed LNI 192.168.204.146@tcp18 [10513.057304] LNet: Removed LNI 192.168.204.146@tcp19 [10513.059173] LNet: Removed LNI 192.168.204.146@tcp20 [10513.060821] LNet: Removed LNI 192.168.204.146@tcp21 [10513.062400] LNet: Removed LNI 192.168.204.146@tcp22 [10513.063930] LNet: Removed LNI 192.168.204.146@tcp23 [10513.065537] LNet: Removed LNI 192.168.204.146@tcp24 [10513.067054] LNet: Removed LNI 192.168.204.146@tcp25 [10513.068475] LNet: Removed LNI 192.168.204.146@tcp26 [10513.069981] LNet: Removed LNI 192.168.204.146@tcp27 [10513.071552] LNet: Removed LNI 192.168.204.146@tcp28 [10513.073101] LNet: Removed LNI 192.168.204.146@tcp29 [10513.074535] LNet: Removed LNI 192.168.204.146@tcp30 [10513.075997] LNet: Removed LNI 192.168.204.146@tcp31 [10513.077526] LNet: Removed LNI 192.168.204.146@tcp32 [10513.079103] LNet: Removed LNI 192.168.204.146@tcp33 [10513.080489] LNet: Removed LNI 192.168.204.146@tcp34 [10513.082027] LNet: Removed LNI 192.168.204.146@tcp35 [10513.083563] LNet: Removed LNI 192.168.204.146@tcp36 [10513.085031] LNet: Removed LNI 192.168.204.146@tcp37 [10513.086467] LNet: Removed LNI 192.168.204.146@tcp38 [10513.088051] LNet: Removed LNI 192.168.204.146@tcp39 [10513.089521] LNet: Removed LNI 192.168.204.146@tcp40 [10513.090978] LNet: Removed LNI 192.168.204.146@tcp41 [10513.092562] LNet: Removed LNI 192.168.204.146@tcp42 [10513.094124] LNet: Removed LNI 192.168.204.146@tcp43 [10513.095616] LNet: Removed LNI 192.168.204.146@tcp44 [10513.097104] LNet: Removed LNI 192.168.204.146@tcp45 [10513.098542] LNet: Removed LNI 192.168.204.146@tcp46 [10513.099954] LNet: Removed LNI 192.168.204.146@tcp47 [10513.101335] LNet: Removed LNI 192.168.204.146@tcp48 [10513.102842] LNet: Removed LNI 192.168.204.146@tcp49 [10513.110353] LNet: Removed LNI 192.168.204.146@tcp50 [10513.389259] Key type .llcrypt unregistered [10513.390428] Key type ._llcrypt unregistered [10521.316435] Key type ._llcrypt registered [10521.317884] Key type .llcrypt registered [10521.365710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10528.769482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10529.102371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10529.139993] alg: No test for adler32 (adler32-zlib) [10530.001760] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10530.104067] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10531.688129] Key type lgssc registered [10532.044645] Lustre: Echo OBD driver; http://www.lustre.org/ [10546.097037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10546.105460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.220603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10547.238495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10547.296323] Lustre: lustre-MDT0000: new disk, initializing [10547.331150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10547.340076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10549.047407] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10553.889040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.943710] Lustre: 504842:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10553.982542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10553.985631] Lustre: Skipped 1 previous similar message [10554.028681] Lustre: lustre-MDT0001: new disk, initializing [10554.082006] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10554.104146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10554.110882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10555.596787] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10559.905206] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10562.169384] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10564.991683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10565.089569] Lustre: *** cfs_fail_loc=90f, val=0*** [10565.125209] Lustre: lustre-OST0000: new disk, initializing [10565.128665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10565.133821] Lustre: 506860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10565.174253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10566.972896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10566.976343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10567.010971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10567.330439] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10571.056293] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10577.379924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10577.381545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10577.387576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10577.392464] Lustre: Skipped 1 previous similar message [10579.545665] Lustre: server umount lustre-OST0000 complete [10582.497853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.498355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.502136] Lustre: Skipped 2 previous similar messages [10582.505540] Lustre: Skipped 1 previous similar message [10587.617041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.619331] Lustre: Skipped 1 previous similar message [10592.737024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10592.739106] Lustre: Skipped 1 previous similar message [10595.296107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10595.374798] Lustre: server umount lustre-MDT0000 complete [10596.906140] LustreError: 504835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148135 with bad export cookie 7418491286569868566 [10596.914363] LustreError: 504835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10596.915783] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10597.073702] Lustre: server umount lustre-MDT0001 complete [10600.848533] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10602.027234] Key type lgssc unregistered [10602.171685] LNet: 508764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10602.177120] LNetError: 508764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10602.190581] LNet: Removed LNI 192.168.204.146@tcp [10602.545297] Key type .llcrypt unregistered [10602.547369] Key type ._llcrypt unregistered [10610.178464] Key type ._llcrypt registered [10610.179907] Key type .llcrypt registered [10610.218741] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10610.523062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10610.531074] alg: No test for adler32 (adler32-zlib) [10611.386274] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10611.470400] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10613.048125] Key type lgssc registered [10613.421474] Lustre: Echo OBD driver; http://www.lustre.org/ [10615.912786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10615.921428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.065352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10618.367541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10621.479020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10621.619753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10622.915284] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10625.184037] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10627.058626] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10629.457821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10629.573542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10631.497904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10633.946526] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10648.642984] loop0: detected capacity change from 0 to 204800000 [10648.718510] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10648.725031] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10648.729561] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10651.626815] loop0: detected capacity change from 0 to 204800000 [10651.750427] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10651.755888] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10651.763116] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10654.584702] loop0: detected capacity change from 0 to 204800000 [10657.264336] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10657.335780] Lustre: fs15246-OST0000: new disk, initializing [10657.338573] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10657.344196] Lustre: 513626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10657.379625] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10660.116216] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10665.295755] loop1: detected capacity change from 0 to 204800000 [10668.278500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.335508] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10668.365000] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10668.408431] Lustre: fs15246-MDT0000: new disk, initializing [10668.500778] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10668.519377] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10668.560422] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10669.603133] Lustre: 514746:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000696126a1 x1869766353520384/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783148223 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10670.837930] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10678.698837] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10678.733683] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10678.801689] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10708.995763] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10709.005334] LustreError: 510007:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10709.015364] LustreError: 510007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10709.373296] Lustre: server umount fs15246-MDT0000 complete [10714.092402] LustreError: 512500:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10719.257238] LustreError: 511478:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10724.326830] LustreError: 511478:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.441785] LustreError: 511478:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10733.379250] Lustre: server umount fs15246-OST0000 complete [10756.577333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10756.581601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.582266] LustreError: Skipped 1 previous similar message [10756.583683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10756.612842] Lustre: Skipped 1 previous similar message [10760.163454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10760.186398] Lustre: Skipped 2 previous similar messages [10765.290888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10765.339251] Lustre: Skipped 1 previous similar message [10769.888297] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10770.429985] Lustre: server umount lustre-OST0000 complete [10775.522792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.525437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10775.542203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.572087] Lustre: Skipped 1 previous similar message [10780.643173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.657026] Lustre: Skipped 2 previous similar messages [10782.068645] Lustre: server umount lustre-MDT0000 complete [10785.763122] LustreError: 510006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.792807] LustreError: 510006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10788.533455] LustreError: 509993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148326 with bad export cookie 15609414954100376186 [10788.550821] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10788.565832] LustreError: 509993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10789.745551] Lustre: server umount lustre-MDT0001 complete [10805.865749] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10809.305710] Key type lgssc unregistered [10809.733287] LNet: 517849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10809.743816] LNetError: 517849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10810.790619] LNet: Removed LNI 192.168.204.146@tcp [10812.118262] Key type .llcrypt unregistered [10812.121929] Key type ._llcrypt unregistered [10833.572282] Key type ._llcrypt registered [10833.577074] Key type .llcrypt registered [10833.773826] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10834.862765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10835.040075] alg: No test for adler32 (adler32-zlib) [10836.164866] Lustre: Lustre: Build Version: 2.17.54_83_gca256e9 [10836.481694] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10838.192468] Key type lgssc registered [10839.302365] Lustre: Echo OBD driver; http://www.lustre.org/ [10875.848986] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10886.182924] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10924.665631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10924.703607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.915412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10925.948658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10926.032191] Lustre: lustre-MDT0000: new disk, initializing [10926.169312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10926.192156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10931.195629] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10941.948467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.067073] Lustre: 522329:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10942.149444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10942.153401] Lustre: Skipped 1 previous similar message [10942.162749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10942.229547] Lustre: lustre-MDT0001: new disk, initializing [10942.309438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10942.345748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10942.356428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10946.841235] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10955.548737] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10961.786498] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10968.699420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.129639] Lustre: lustre-OST0000: new disk, initializing [10969.139968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10969.156364] Lustre: 524297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10969.272887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10975.893529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10976.922891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10976.943305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10977.077254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10986.302871] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10992.177290] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10992.457667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10997.904876] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10998.184168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11002.856077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11002.858613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11002.889942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11002.897770] Lustre: Skipped 1 previous similar message [11006.747936] Lustre: server umount lustre-OST0000 complete [11013.090322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.101532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11013.111768] Lustre: Skipped 2 previous similar messages [11013.132171] Lustre: Skipped 1 previous similar message [11018.031707] Lustre: server umount lustre-MDT0000 complete [11021.802663] LustreError: 522322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783148559 with bad export cookie 13915901081109876283 [11021.809347] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.814913] LustreError: 522322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11022.453855] Lustre: server umount lustre-MDT0001 complete [11050.077832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11062.102831] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11102.732775] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10762 sec ========= 03:04:00 (1783148640) [11104.376442] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:04:02 (1783148642) === [11107.733170] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:04:05 (1783148645) === [11156.196057] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11161.382473] Key type lgssc unregistered [11161.801615] LNet: 530946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11161.818900] LNetError: 530946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11161.855365] LNet: Removed LNI 192.168.204.146@tcp [11163.369223] Key type .llcrypt unregistered [11163.372278] Key type ._llcrypt unregistered