[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508806902 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003032] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010167] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.022129] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027751] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030141] *** VALIDATE tmpfs *** [ 0.031133] *** VALIDATE proc *** [ 0.033019] *** VALIDATE cgroup *** [ 0.034000] *** VALIDATE cgroup2 *** [ 0.034289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041132] debug: unmapping init [mem 0xffffffffaf859000-0xffffffffaf860fff] [ 0.044107] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045786] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053301] rcu: Hierarchical SRCU implementation. [ 0.055456] smp: Bringing up secondary CPUs ... [ 0.056551] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.060367] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134035] node 0 deferred pages initialised in 68ms [ 0.137105] devtmpfs: initialized [ 0.138205] x86/mm: Memory block size: 128MB [ 0.140483] gcov: version magic: 0x41383552 [ 0.141656] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148335] pinctrl core: initialized pinctrl subsystem [ 0.150165] [ 0.150711] ************************************************************* [ 0.152011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154011] ** ** [ 0.155010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157010] ** ** [ 0.159008] ** This means that this kernel is built to expose internal ** [ 0.160006] ** IOMMU data structures, which may compromise security on ** [ 0.162011] ** your system. ** [ 0.164012] ** ** [ 0.167016] ** If you see this message and you are not debugging the ** [ 0.169010] ** kernel, report this immediately to your vendor! ** [ 0.171011] ** ** [ 0.173013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175015] ************************************************************* [ 0.177622] NET: Registered protocol family 16 [ 0.179398] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188361] cpuidle: using governor menu [ 0.189628] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192118] PCI: Using configuration type 1 for base access [ 0.194120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208155] cryptd: max_cpu_qlen set to 1000 [ 0.210026] ACPI: Added _OSI(Module Device) [ 0.212013] ACPI: Added _OSI(Processor Device) [ 0.213013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219852] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226575] ACPI: Interpreter enabled [ 0.228128] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230013] ACPI: Using IOAPIC for interrupt routing [ 0.231122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235376] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245187] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260353] acpiphp: Slot [2] registered [ 0.261125] acpiphp: Slot [5] registered [ 0.263119] acpiphp: Slot [6] registered [ 0.265111] acpiphp: Slot [7] registered [ 0.266136] acpiphp: Slot [8] registered [ 0.268122] acpiphp: Slot [9] registered [ 0.270122] acpiphp: Slot [10] registered [ 0.271088] acpiphp: Slot [3] registered [ 0.273077] acpiphp: Slot [4] registered [ 0.274108] acpiphp: Slot [11] registered [ 0.275092] acpiphp: Slot [12] registered [ 0.275937] acpiphp: Slot [13] registered [ 0.277091] acpiphp: Slot [14] registered [ 0.279122] acpiphp: Slot [15] registered [ 0.281087] acpiphp: Slot [16] registered [ 0.282151] acpiphp: Slot [17] registered [ 0.283101] acpiphp: Slot [18] registered [ 0.285108] acpiphp: Slot [19] registered [ 0.286090] acpiphp: Slot [20] registered [ 0.288113] acpiphp: Slot [21] registered [ 0.289121] acpiphp: Slot [22] registered [ 0.291110] acpiphp: Slot [23] registered [ 0.293105] acpiphp: Slot [24] registered [ 0.294093] acpiphp: Slot [25] registered [ 0.295149] acpiphp: Slot [26] registered [ 0.297082] acpiphp: Slot [27] registered [ 0.298109] acpiphp: Slot [28] registered [ 0.299068] acpiphp: Slot [29] registered [ 0.301094] acpiphp: Slot [30] registered [ 0.302083] acpiphp: Slot [31] registered [ 0.303093] PCI host bridge to bus 0000:00 [ 0.305019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315057] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318913] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322229] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332792] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.337094] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.341013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346493] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348655] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.361024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.388267] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.419017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430363] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.438020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.447019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.461017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475397] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.482017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518885] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.576242] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.584013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.594019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.614016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.624614] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.632018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.638016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.656017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669191] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.672343] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.673304] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.675358] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677181] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.684058] iommu: Default domain type: Passthrough [ 0.686292] SCSI subsystem initialized [ 0.688179] ACPI: bus type USB registered [ 0.689087] usbcore: registered new interface driver usbfs [ 0.690073] usbcore: registered new interface driver hub [ 0.692090] usbcore: registered new device driver usb [ 0.694192] pps_core: LinuxPPS API ver. 1 registered [ 0.695009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.698069] PTP clock support registered [ 0.700036] EDAC MC: Ver: 3.0.0 [ 0.702148] PCI: Using ACPI for IRQ routing [ 0.703629] NetLabel: Initializing [ 0.705013] NetLabel: domain hash size = 128 [ 0.707011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.709078] NetLabel: unlabeled traffic allowed by default [ 0.711158] vgaarb: loaded [ 0.713301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.716015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721375] clocksource: Switched to clocksource kvm-clock [ 0.826297] VFS: Disk quotas dquot_6.6.0 [ 0.828215] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.831312] *** VALIDATE ramfs *** [ 0.832845] *** VALIDATE hugetlbfs *** [ 0.834886] pnp: PnP ACPI init [ 0.838208] pnp: PnP ACPI: found 6 devices [ 0.856168] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860057] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.862706] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.865145] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.868402] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.871542] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.875275] NET: Registered protocol family 2 [ 0.878544] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884400] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.888136] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893548] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897312] TCP: Hash tables configured (established 65536 bind 65536) [ 0.901162] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.904662] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908129] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911256] NET: Registered protocol family 1 [ 0.914146] RPC: Registered named UNIX socket transport module. [ 0.916166] RPC: Registered udp transport module. [ 0.917780] RPC: Registered tcp transport module. [ 0.919681] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.922178] NET: Registered protocol family 44 [ 0.924040] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.926315] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.928892] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.931525] PCI: CLS 0 bytes, default 64 [ 0.933290] Unpacking initramfs... [ 2.324966] debug: unmapping init [mem 0xffff98107cc54000-0xffff98107ffbffff] [ 2.330033] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.332859] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.336268] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.841123] Initialise system trusted keyrings [ 2.842877] Key type blacklist registered [ 2.844602] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.854048] zbud: loaded [ 2.857546] *** VALIDATE nfs *** [ 2.858889] *** VALIDATE nfs4 *** [ 2.860540] pstore: using deflate compression [ 2.863958] Platform Keyring initialized [ 2.961770] NET: Registered protocol family 38 [ 2.963362] Key type asymmetric registered [ 2.964989] Asymmetric key parser 'x509' registered [ 2.966627] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.970417] io scheduler mq-deadline registered [ 2.971709] io scheduler kyber registered [ 2.973187] io scheduler bfq registered [ 2.974943] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.977428] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.979675] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.982163] ACPI: Power Button [PWRF] [ 2.986810] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.993432] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.015545] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.024402] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.038232] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.065319] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.091914] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.096516] Non-volatile memory driver v1.3 [ 3.098443] Linux agpgart interface v0.103 [ 3.129645] virtio_blk virtio1: [vda] 149456 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.132793] vda: detected capacity change from 0 to 76521472 [ 3.152356] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.155294] vdb: detected capacity change from 0 to 1073741824 [ 3.170406] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.173323] vdc: detected capacity change from 0 to 2621440000 [ 3.187783] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.190760] vdd: detected capacity change from 0 to 2621440000 [ 3.204240] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.207183] vde: detected capacity change from 0 to 4294967296 [ 3.227725] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.231030] vdf: detected capacity change from 0 to 4294967296 [ 3.237835] libphy: Fixed MDIO Bus: probed [ 3.244477] usbcore: registered new interface driver usbserial_generic [ 3.247139] usbserial: USB Serial support registered for generic [ 3.249549] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.253554] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.255364] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.257728] mousedev: PS/2 mouse device common for all mice [ 3.260022] rtc_cmos 00:05: RTC can wake from S4 [ 3.262358] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.265910] rtc_cmos 00:05: registered as rtc0 [ 3.268504] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.268801] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.271848] intel_pstate: CPU model not supported [ 3.276416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.279661] hid: raw HID events driver (C) Jiri Kosina [ 3.281682] usbcore: registered new interface driver usbhid [ 3.283549] usbhid: USB HID core driver [ 3.285215] drop_monitor: Initializing network drop monitor service [ 3.287661] Initializing XFRM netlink socket [ 3.289176] NET: Registered protocol family 10 [ 3.291826] Segment Routing with IPv6 [ 3.292902] NET: Registered protocol family 17 [ 3.294538] mpls_gso: MPLS GSO support [ 3.299237] RAS: Correctable Errors collector initialized. [ 3.301176] AVX version of gcm_enc/dec engaged. [ 3.302474] AES CTR mode by8 optimization enabled [ 3.367866] sched_clock: Marking stable (3367843764, 0)->(4378059969, -1010216205) [ 3.370760] registered taskstats version 1 [ 3.372552] Loading compiled-in X.509 certificates [ 3.374118] zswap: loaded using pool lzo/zbud [ 3.395613] Key type big_key registered [ 3.406645] Key type encrypted registered [ 3.408575] ima: No TPM chip found, activating TPM-bypass! [ 3.410608] ima: Allocated hash algorithm: sha1 [ 3.412297] ima: No architecture policies found [ 3.413808] evm: Initialising EVM extended attributes: [ 3.415546] evm: security.selinux [ 3.416530] evm: security.ima [ 3.417073] evm: security.capability [ 3.418036] evm: HMAC attrs: 0x1 [ 3.420072] rtc_cmos 00:05: setting system clock to 2026-08-22 07:25:30 UTC (1787383530) [ 3.426313] debug: unmapping init [mem 0xffffffffb0803000-0xffffffffb09fffff] [ 3.429101] debug: unmapping init [mem 0xffffffffaf582000-0xffffffffaf858fff] [ 3.437078] Write protecting the kernel read-only data: 28672k [ 3.440530] debug: unmapping init [mem 0xffffffffadc03000-0xffffffffaddfffff] [ 3.443375] debug: unmapping init [mem 0xffffffffae514000-0xffffffffae5fffff] [ 3.473991] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.480638] systemd[1]: Detected virtualization kvm. [ 3.482086] systemd[1]: Detected architecture x86-64. [ 3.483479] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.508766] systemd[1]: No hostname configured. [ 3.510023] systemd[1]: Set hostname to . [ 3.512158] random: systemd: uninitialized urandom read (16 bytes read) [ 3.514577] systemd[1]: Initializing machine ID from random generator. [ 3.552833] random: ln: uninitialized urandom read (6 bytes read) [ 3.637798] random: systemd: uninitialized urandom read (16 bytes read) [ 3.640608] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.645247] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.649668] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.223131] device-mapper: uevent: version 1.0.3 [ 4.225491] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.893523] virtio_net virtio0 ens2: renamed from eth0 [ 4.926485] random: fast init done [ 4.967284] scsi host0: ata_piix [ 4.987751] scsi host1: ata_piix [ 5.049662] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.051763] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.643273] dracut-initqueue[600]: RTNETLINK answers: File exists [ 9.928985] random: crng init done [ 9.930154] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.300985] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.582688] printk: systemd: 21 output lines suppressed due to ratelimiting [ 11.841629] SELinux: Disabled at runtime. [ 11.907747] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.915820] systemd[1]: Detected virtualization kvm. [ 11.916992] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.415677] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.418442] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.423449] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.427303] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.430446] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.438259] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.442591] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ 12.557427] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.931594] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.343175] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.367482] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.487789] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.507296] EDAC sbridge: Ver: 1.1.2 [ 15.363164] Key type dns_resolver registered [ 15.671712] NFS: Registering the id_resolver key type [ 15.673736] Key type id_resolver registered [ 15.675585] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 70.839759] libcfs: loading out-of-tree module taints kernel. [ 70.890436] Key type ._llcrypt registered [ 70.892958] Key type .llcrypt registered [ 70.999903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 87.575305] hrtimer: interrupt took 4272159 ns [ 89.433746] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 91.153764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.166854] alg: No test for adler32 (adler32-zlib) [ 92.668118] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 93.739346] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 95.576835] Key type lgssc registered [ 97.686108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.857755] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 157.989194] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 171.860675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.905532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.300275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.346950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.505722] Lustre: lustre-MDT0000: new disk, initializing [ 173.640222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 173.680785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.142399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.903450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.008376] Lustre: 6528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.074776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.079096] Lustre: Skipped 1 previous similar message [ 192.192600] Lustre: lustre-MDT0001: new disk, initializing [ 192.308672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.363389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 192.371336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.421309] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.247716] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.588233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.903962] Lustre: lustre-OST0000: new disk, initializing [ 210.911366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.918052] Lustre: 8466:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.996555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.547191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 211.564646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 211.613887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.412388] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.030994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.161477] Lustre: lustre-OST0001: new disk, initializing [ 232.165176] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 232.171449] Lustre: 9537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.233719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.234808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.687314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 240.697532] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 240.857567] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.699366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.610660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.373414] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.881323] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 276.948558] Lustre: DEBUG MARKER: Client: 2.17.54.113 [ 279.283796] Lustre: DEBUG MARKER: MDS: 2.17.54.113 [ 281.655451] Lustre: DEBUG MARKER: OSS: 2.17.54.113 [ 283.476117] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 03:30:08 EDT 2026 [ 299.552963] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 300.855277] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 317.924262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.931104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 317.942373] Lustre: Skipped 3 previous similar messages [ 317.953119] Lustre: Skipped 2 previous similar messages [ 322.756699] Lustre: server umount lustre-MDT0000 complete [ 327.129194] LustreError: 7477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787383854 with bad export cookie 9747978200580670061 [ 327.133386] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.139081] LustreError: 7477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 328.161462] LustreError: 10112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.162091] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.165335] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.185497] LustreError: 10112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 328.218113] Lustre: Skipped 1 previous similar message [ 332.192923] LustreError: 10112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.196323] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.237542] Lustre: Skipped 2 previous similar messages [ 333.719916] Lustre: server umount lustre-MDT0001 complete [ 338.403759] Lustre: server umount lustre-OST0000 complete [ 342.724718] Lustre: server umount lustre-OST0001 complete [ 349.454687] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 359.046636] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 395.021954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.254633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.277776] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.361346] Lustre: lustre-MDT0000: new disk, initializing [ 395.437556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.457315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.842943] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 412.151925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.257442] Lustre: 15098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.327286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.335633] Lustre: Skipped 1 previous similar message [ 412.441384] Lustre: lustre-MDT0001: new disk, initializing [ 412.512823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.559139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.589793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.088974] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 430.398021] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 437.372535] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 444.508038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 444.778977] Lustre: lustre-OST0000: new disk, initializing [ 444.785993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 444.801239] Lustre: 17068:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 444.885652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.351434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.369859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.528286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 451.890647] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 461.916549] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 466.617724] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 466.789220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.945684] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.232625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.155042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.158520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.172996] LustreError: Skipped 1 previous similar message [ 477.175341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.199987] Lustre: Skipped 1 previous similar message [ 480.505664] Lustre: server umount lustre-OST0000 complete [ 487.394595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.407476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 487.432621] Lustre: Skipped 1 previous similar message [ 487.438149] Lustre: Skipped 1 previous similar message [ 490.871747] Lustre: server umount lustre-MDT0000 complete [ 492.514217] LustreError: 15110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.530700] LustreError: 15110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 494.786054] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384021 with bad export cookie 9747978200580671594 [ 494.792057] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.799118] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 495.241936] Lustre: server umount lustre-MDT0001 complete [ 499.783502] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:33:45 (1787384025) [ 502.484039] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 504.605598] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:33:49 (1787384029) [ 511.810117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.433893] LustreError: 19476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.544711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.176425] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 517.601580] LustreError: 19477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.695091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.230566] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 539.053454] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 544.851173] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 551.172481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 558.251564] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 565.591815] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 573.409950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.412395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.425084] LustreError: Skipped 1 previous similar message [ 573.429737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.429746] Lustre: Skipped 1 previous similar message [ 573.467950] Lustre: Skipped 1 previous similar message [ 578.882644] Lustre: server umount lustre-OST0000 complete [ 583.650768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.668338] Lustre: Skipped 1 previous similar message [ 589.172851] LustreError: 22254:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9810c7566400) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 591.273374] Lustre: server umount lustre-MDT0000 complete [ 593.892654] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.919834] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 595.140980] LustreError: 19458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384122 with bad export cookie 9747978200580672476 [ 595.156674] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.452765] Lustre: server umount lustre-MDT0001 complete [ 605.677874] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 608.738929] Key type lgssc unregistered [ 609.054231] LNet: 23007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.065256] LNetError: 23007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.097530] LNet: Removed LNI 192.168.204.144@tcp [ 609.901226] Key type .llcrypt unregistered [ 609.907901] Key type ._llcrypt unregistered [ 633.746618] Key type ._llcrypt registered [ 633.749505] Key type .llcrypt registered [ 633.851315] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 634.938694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.235307] alg: No test for adler32 (adler32-zlib) [ 636.427540] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 636.695099] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 638.448859] Key type lgssc registered [ 639.865810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 648.804714] loop: module loaded [ 648.871132] loop0: detected capacity change from 0 to 204800000 [ 649.075973] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.087862] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.282329] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.108243] loop0: detected capacity change from 0 to 204800000 [ 655.297475] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.312104] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.327401] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 660.454777] loop0: detected capacity change from 0 to 204800000 [ 665.958622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 665.986111] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.300642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 667.417623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 667.565994] Lustre: lustre-MDT0000: new disk, initializing [ 667.696720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.734812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 672.298052] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 680.502327] loop1: detected capacity change from 0 to 204800000 [ 685.669318] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 685.875366] Lustre: lustre-OST07c6: new disk, initializing [ 685.881579] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 685.891784] Lustre: Skipped 1 previous similar message [ 685.901458] Lustre: 26198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 685.975394] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 691.598428] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 693.359432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 693.372623] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 693.415499] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 695.983215] Lustre: 25182:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 695.993676] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 696.002648] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 696.009688] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 696.017284] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 696.027574] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 714.241327] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 714.252685] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 714.258100] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 714.264648] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 714.281839] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 714.300481] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.558559] Lustre: 25182:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 719.569258] Lustre: 25182:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.577184] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.584180] Lustre: 25182:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.592069] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.598053] Lustre: 25182:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.607346] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 719.616303] Lustre: 25182:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.620181] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 719.623299] Lustre: 25182:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.627644] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.632169] Lustre: 25182:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.774490] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 722.801903] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.815244] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 722.819582] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.831887] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.838240] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.844344] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 722.851136] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.861565] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 722.869749] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.875609] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.887265] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.389426] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 727.399542] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 727.407291] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.414649] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.420048] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.431057] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.438495] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 727.453624] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.465262] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 727.472420] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 727.482614] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 727.491029] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 738.744854] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 738.754698] Lustre: 25183:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 738.760818] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 738.770577] Lustre: 25183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.776367] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 738.784350] Lustre: 25183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.796963] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 738.803188] Lustre: 25183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.808665] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 738.813493] Lustre: 25183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 738.819332] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 738.828746] Lustre: 25183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.500433] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.516512] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 790.976214] Lustre: server umount lustre-OST07c6 complete [ 800.512316] Lustre: server umount lustre-MDT0000 complete [ 817.027583] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 819.710355] Key type lgssc unregistered [ 820.030183] LNet: 28749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.039340] LNetError: 28749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.059863] LNet: Removed LNI 192.168.204.144@tcp [ 820.845710] Key type .llcrypt unregistered [ 820.848227] Key type ._llcrypt unregistered [ 842.606701] Key type ._llcrypt registered [ 842.612826] Key type .llcrypt registered [ 842.796120] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 844.837678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.972951] alg: No test for adler32 (adler32-zlib) [ 845.925781] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 846.143741] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 847.920376] Key type lgssc registered [ 849.130563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.548215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.572526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.958765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.999554] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 867.297959] LustreError: 30025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.310098] LustreError: 30025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 869.878272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.528331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 875.874492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 884.585546] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 889.460809] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.501729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 896.876680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.029224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 902.506700] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 909.179392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 927.715405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.729398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.743979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.788673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.791895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.800451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.223850] Lustre: server umount lustre-OST0000 complete [ 941.031930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.036429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.038913] Lustre: Skipped 1 previous similar message [ 941.065554] Lustre: Skipped 2 previous similar messages [ 944.040628] Lustre: server umount lustre-MDT0000 complete [ 946.147489] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.172570] LustreError: 30024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 948.670989] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384475 with bad export cookie 209316633935194679 [ 948.687769] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 948.690409] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 949.043857] Lustre: server umount lustre-MDT0001 complete [ 958.855293] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 961.617365] Key type lgssc unregistered [ 961.879303] LNet: 33761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.886501] LNetError: 33761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.900521] LNet: Removed LNI 192.168.204.144@tcp [ 962.696209] Key type .llcrypt unregistered [ 962.699732] Key type ._llcrypt unregistered [ 981.628279] Key type ._llcrypt registered [ 981.635471] Key type .llcrypt registered [ 981.764974] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 982.552231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 982.623883] alg: No test for adler32 (adler32-zlib) [ 983.894960] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 984.102697] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 985.768199] Key type lgssc registered [ 986.829776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 994.610547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 994.633598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.283853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1001.494469] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1006.563454] LustreError: 35033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.577586] LustreError: 35033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1011.684081] LustreError: 35034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.890976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.384247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.536387] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1027.838821] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.206295] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1042.931979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.294403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.647094] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1057.898200] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1067.895532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1083.367896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1083.368658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1083.404748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1083.427178] Lustre: Skipped 1 previous similar message [ 1085.156274] Lustre: server umount lustre-OST0000 complete [ 1093.619499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.625618] Lustre: Skipped 1 previous similar message [ 1093.631635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1093.640487] Lustre: Skipped 1 previous similar message [ 1098.721352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.732538] Lustre: Skipped 1 previous similar message [ 1103.842321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.854496] Lustre: Skipped 1 previous similar message [ 1104.352215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1104.657796] Lustre: server umount lustre-MDT0000 complete [ 1108.864687] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384635 with bad export cookie 5984433532750219890 [ 1108.868615] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.882952] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.279394] Lustre: server umount lustre-MDT0001 complete [ 1120.196884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1123.145263] Key type lgssc unregistered [ 1123.408467] LNet: 38591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.420154] LNetError: 38591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.440055] LNet: Removed LNI 192.168.204.144@tcp [ 1124.367351] Key type .llcrypt unregistered [ 1124.369299] Key type ._llcrypt unregistered [ 1151.815765] Key type ._llcrypt registered [ 1151.818408] Key type .llcrypt registered [ 1152.013172] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1153.152787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1153.186069] alg: No test for adler32 (adler32-zlib) [ 1154.383439] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1154.693192] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1156.352896] Key type lgssc registered [ 1157.482764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1165.104951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1165.127633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.637346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1171.225268] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1176.037054] LustreError: 40008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.064692] LustreError: 40008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1180.540154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1180.944614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1185.665498] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1193.806910] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1199.651654] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1207.814612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.112085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.201114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1214.724682] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1222.826384] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1230.652247] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1230.818894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.841023] Lustre: Skipped 1 previous similar message [ 1235.748765] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1235.757354] Lustre: Skipped 3 previous similar messages [ 1235.967318] LustreError: 40008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.998380] LustreError: 40008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1236.067836] Lustre: server umount lustre-MDT0000 complete [ 1240.202942] LustreError: 41852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384767 with bad export cookie 5706193089199756390 [ 1240.207546] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.213338] LustreError: 41852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1240.867857] LustreError: 40004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.951320] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 1241.059045] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.090968] Lustre: Skipped 1 previous similar message [ 1245.092453] LustreError: 41736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.095232] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1245.127432] LustreError: 41736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1245.153672] Lustre: Skipped 1 previous similar message [ 1246.833366] Lustre: server umount lustre-MDT0001 complete [ 1276.384172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1276.600462] Lustre: server umount lustre-OST0000 complete [ 1293.981772] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1297.075851] Key type lgssc unregistered [ 1297.432842] LNet: 43877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.444470] LNetError: 43877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.468745] LNet: Removed LNI 192.168.204.144@tcp [ 1298.470393] Key type .llcrypt unregistered [ 1298.475593] Key type ._llcrypt unregistered [ 1322.642224] Key type ._llcrypt registered [ 1322.647767] Key type .llcrypt registered [ 1322.769666] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1324.267289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.384488] alg: No test for adler32 (adler32-zlib) [ 1325.550421] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1325.829300] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1327.552988] Key type lgssc registered [ 1329.064081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.571261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1338.605515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.323752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.915467] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1350.632014] LustreError: 45297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.656455] LustreError: 45297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1355.047371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.498616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.927975] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1369.251936] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1374.749648] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1382.469821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1382.798972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.919065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1389.303155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1397.839480] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1405.415056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.435119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.535684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.540286] Lustre: Skipped 3 previous similar messages [ 1411.347322] Lustre: server umount lustre-OST0000 complete [ 1432.547775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1432.559985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.578608] Lustre: Skipped 1 previous similar message [ 1432.596105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.606910] Lustre: Skipped 1 previous similar message [ 1435.683586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1435.694364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.584877] Lustre: server umount lustre-MDT0000 complete [ 1440.739451] LustreError: 45292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.759548] LustreError: 45292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1441.694166] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787384968 with bad export cookie 5623986065740068460 [ 1441.697184] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.720105] LustreError: 45278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1448.173720] Lustre: server umount lustre-MDT0001 complete [ 1461.390228] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1464.435384] Key type lgssc unregistered [ 1464.893267] LNet: 49005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.897786] LNetError: 49005:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.932880] LNet: Removed LNI 192.168.204.144@tcp [ 1465.897758] Key type .llcrypt unregistered [ 1465.902836] Key type ._llcrypt unregistered [ 1492.720823] Key type ._llcrypt registered [ 1492.727925] Key type .llcrypt registered [ 1492.866619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1493.883121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1493.907194] alg: No test for adler32 (adler32-zlib) [ 1495.143681] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1495.356802] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1497.024867] Key type lgssc registered [ 1498.360725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1506.367651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1506.409065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.919276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.403718] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1517.028938] LustreError: 50424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.055480] LustreError: 50424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1520.388830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.832107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1525.517892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1532.828714] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1538.338528] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1545.718681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1546.046475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.180558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1552.428291] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1559.835208] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1569.308360] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1573.120172] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1585.439289] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 3h51m50s [ 1587.111946] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1591.265911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.274320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.289562] Lustre: Skipped 1 previous similar message [ 1592.362411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.375026] Lustre: Skipped 4 previous similar messages [ 1594.210651] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1594.224917] Lustre: Skipped 1 previous similar message [ 1595.410654] Lustre: server umount lustre-MDT0000 complete [ 1598.449437] LustreError: 50425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.468590] LustreError: 50425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1598.960914] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1600.290363] LustreError: 51857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.579172] LustreError: 50424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.630446] LustreError: 50424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1605.693786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.806605] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.031621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.088566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.096666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.946521] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1615.036034] Lustre: DEBUG MARKER: Wait for df (36524) ... [ 1617.217780] Lustre: DEBUG MARKER: done [ 1621.989259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1621.997183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.014315] Lustre: Skipped 1 previous similar message [ 1622.023264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.620051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.627304] Lustre: Skipped 3 previous similar messages [ 1627.807635] Lustre: server umount lustre-OST0000 complete [ 1632.739187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.739601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.769890] Lustre: Skipped 2 previous similar messages [ 1637.799925] Lustre: server umount lustre-MDT0000 complete [ 1638.885468] LustreError: 51436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.908853] LustreError: 51436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1642.109565] LustreError: 50405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385169 with bad export cookie 16874714354010861944 [ 1642.117844] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.126101] LustreError: 50405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1642.571461] Lustre: server umount lustre-MDT0001 complete [ 1653.820621] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1656.535555] Key type lgssc unregistered [ 1656.771656] LNet: 55347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.782416] LNetError: 55347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.808695] LNet: Removed LNI 192.168.204.144@tcp [ 1657.640818] Key type .llcrypt unregistered [ 1657.642430] Key type ._llcrypt unregistered [ 1697.254432] Key type ._llcrypt registered [ 1697.258769] Key type .llcrypt registered [ 1697.364567] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1698.731490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.748893] alg: No test for adler32 (adler32-zlib) [ 1699.860707] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1700.159354] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1701.864186] Key type lgssc registered [ 1703.226608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.712513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1710.735562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1712.005773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1712.056929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1712.294048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.190372] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1725.884921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.989511] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1726.016340] Lustre: 57564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1726.259747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1731.003441] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1738.107858] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1743.955763] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1751.204236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1751.456069] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1751.709116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.025621] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1758.352981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1765.739205] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1773.518225] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 3h54m58s [ 1775.307145] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1783.104877] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1785.835571] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1785.887868] LustreError: 57572:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1806.167051] Lustre: lustre-MDT0000: Client cbd29ceb-7158-4665-bae1-87bf684e5805 (at 192.168.204.44@tcp) reconnecting [ 1825.896177] LustreError: 57572:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1827.782947] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1829.491854] Lustre: DEBUG MARKER: done [ 1834.977264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.986444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.997845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.492218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1835.504460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.506383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1835.531414] Lustre: Skipped 1 previous similar message [ 1840.222536] Lustre: server umount lustre-OST0000 complete [ 1844.195230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1844.204648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.218163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.861558] Lustre: server umount lustre-MDT0000 complete [ 1853.409495] LustreError: 59428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385380 with bad export cookie 14183417139189477384 [ 1853.423765] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.703679] Lustre: server umount lustre-MDT0001 complete [ 1863.806036] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1866.791767] Key type lgssc unregistered [ 1867.241812] LNet: 61892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.247222] LNetError: 61892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.279858] LNet: Removed LNI 192.168.204.144@tcp [ 1868.559800] Key type .llcrypt unregistered [ 1868.566420] Key type ._llcrypt unregistered [ 1911.730881] Key type ._llcrypt registered [ 1911.733314] Key type .llcrypt registered [ 1911.877652] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1913.052968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1913.177537] alg: No test for adler32 (adler32-zlib) [ 1914.275929] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 1914.543164] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1916.272181] Key type lgssc registered [ 1917.672491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.958355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.984957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.254091] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1922.295433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1922.538058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.370878] Lustre: Failing over lustre-MDT0000 [ 1923.613339] Lustre: server umount lustre-MDT0000 complete [ 1929.608964] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:57:35 (1787385455) [ 1935.460809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.843120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.045207] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1947.786260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.860817] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1947.888838] Lustre: 64693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1948.226464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.184216] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1959.835104] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1964.449118] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1970.919712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1971.274926] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1971.511267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.063530] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1978.467431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1983.462905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.366559] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1994.948382] Lustre: server umount lustre-MDT0000 complete [ 1998.348201] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385525 with bad export cookie 17184459254689367198 [ 1998.348924] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1998.362589] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1998.631638] Lustre: server umount lustre-MDT0001 complete [ 2001.830876] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2006.203364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2014.688129] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385525/real 1787385525] req@0000000090e656fd x1874209476219904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2014.689199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.717307] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2014.859389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.064459] Lustre: 63616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385531/real 1787385531] req@00000000bc63f362 x1874209476220032/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2022.944793] LustreError: 63612:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ab2aeea1 x1874209476221312/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.343876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.479991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2024.288156] Lustre: 63614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385535/real 1787385535] req@0000000007b0a904 x1874209476220416/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2024.325772] Lustre: 63614:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2024.370932] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2028.014811] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2029.472641] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385540/real 1787385540] req@00000000c6b7613d x1874209476220672/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385556 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2029.503131] Lustre: 63615:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2029.552715] LustreError: 66541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.576604] LustreError: 66541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2034.659496] LustreError: 68458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.676354] LustreError: 68458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2036.624497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.149722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2042.346731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2042.389587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2049.554076] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2054.958604] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2061.675547] Lustre: server umount lustre-MDT0000 complete [ 2062.818433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.825834] LustreError: 66542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.840903] Lustre: Skipped 3 previous similar messages [ 2062.865101] LustreError: 66542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2065.529162] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385592 with bad export cookie 17184459254689369480 [ 2065.529981] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.542902] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.832374] Lustre: server umount lustre-MDT0001 complete [ 2072.168536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2082.282850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.322797] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385595/real 1787385595] req@000000000e58a769 x1874209476245888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.385502] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2084.407119] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.594364] LustreError: 63612:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000974f3276 x1874209476247040/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.008483] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.134655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.207667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2096.588567] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2097.120163] Lustre: 63614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787385607/real 1787385607] req@000000003fa1ed63 x1874209476246528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787385623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2097.162966] Lustre: 63614:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2099.697538] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.731417] LustreError: 69193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2103.788397] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2104.376372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.605166] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2109.932963] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2115.083531] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2120.072455] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.402440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.426116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.435140] Lustre: Skipped 2 previous similar messages [ 2130.444385] Lustre: Skipped 2 previous similar messages [ 2135.524035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2135.545425] Lustre: Skipped 2 previous similar messages [ 2140.643274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.658830] Lustre: Skipped 2 previous similar messages [ 2142.177798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2142.414512] Lustre: server umount lustre-MDT0000 complete [ 2145.762996] LustreError: 71139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.776610] LustreError: 71139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2146.531612] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385673 with bad export cookie 17184459254689370859 [ 2146.540692] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.549263] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2147.266604] Lustre: server umount lustre-MDT0001 complete [ 2154.354312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2157.853720] Lustre: DEBUG MARKER: files compared the same [ 2170.569573] Lustre: server umount lustre-OST0000 complete [ 2188.538935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2191.376740] Key type lgssc unregistered [ 2191.668780] LNet: 74430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.684434] LNetError: 74430:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.702902] LNet: Removed LNI 192.168.204.144@tcp [ 2192.568899] Key type .llcrypt unregistered [ 2192.572581] Key type ._llcrypt unregistered [ 2214.939789] Key type ._llcrypt registered [ 2214.942491] Key type .llcrypt registered [ 2215.077742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2216.228724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.315571] alg: No test for adler32 (adler32-zlib) [ 2217.442191] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2217.787670] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2219.536973] Key type lgssc registered [ 2221.119650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.013321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.050620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.521150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.697456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2239.970530] LustreError: 75848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.987667] LustreError: 75848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2241.999723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.296220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2246.381638] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2252.252365] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2257.237110] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2263.064548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.366200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.789710] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 2275.708836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2278.278202] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2286.052905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2286.057742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.068831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2286.093906] Lustre: Skipped 1 previous similar message [ 2291.172607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2291.185894] Lustre: Skipped 1 previous similar message [ 2292.012147] Lustre: server umount lustre-OST0000 complete [ 2296.291964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.296269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2296.311688] Lustre: Skipped 1 previous similar message [ 2296.328018] Lustre: Skipped 2 previous similar messages [ 2301.409352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2301.420552] Lustre: Skipped 1 previous similar message [ 2306.529370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2306.533706] Lustre: Skipped 1 previous similar message [ 2310.113715] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2310.272717] Lustre: server umount lustre-MDT0000 complete [ 2311.648582] LustreError: 75849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.671219] LustreError: 75849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2313.990110] LustreError: 75830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787385841 with bad export cookie 12458811151023451368 [ 2314.004029] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.008106] LustreError: 75830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2314.282605] Lustre: server umount lustre-MDT0001 complete [ 2323.683762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2326.475637] Key type lgssc unregistered [ 2326.780895] LNet: 79378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2326.787522] LNetError: 79378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.828758] LNet: Removed LNI 192.168.204.144@tcp [ 2327.551665] Key type .llcrypt unregistered [ 2327.553564] Key type ._llcrypt unregistered [ 2347.585307] Key type ._llcrypt registered [ 2347.590423] Key type .llcrypt registered [ 2347.680355] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2348.476856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.516553] alg: No test for adler32 (adler32-zlib) [ 2349.545983] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2349.809516] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2351.496252] Key type lgssc registered [ 2352.557542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.916990] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.949134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2375.584363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2380.705178] LustreError: 80752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 2407.392637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.811560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2419.806168] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2426.964439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.119088] Lustre: *** cfs_fail_loc=706, val=0*** [ 2427.621475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.702526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2432.314298] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2432.995227] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.014043] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2438.122517] LustreError: 80778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.400467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.816628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2444.703860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2451.292819] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2456.561951] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2458.599549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.629844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2458.635930] Lustre: Skipped 1 previous similar message [ 2463.718525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.731601] Lustre: Skipped 1 previous similar message [ 2464.814083] Lustre: server umount lustre-OST0000 complete [ 2468.833986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.846171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.856687] Lustre: Skipped 2 previous similar messages [ 2468.886345] Lustre: Skipped 1 previous similar message [ 2473.953871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.961588] Lustre: Skipped 1 previous similar message [ 2474.808735] Lustre: server umount lustre-MDT0000 complete [ 2478.427434] LustreError: 80759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386005 with bad export cookie 4404995677619901215 [ 2478.434278] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.438891] LustreError: 80759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2478.784275] Lustre: server umount lustre-MDT0001 complete [ 2489.829975] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2492.451149] Key type lgssc unregistered [ 2492.783300] LNet: 84366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.788929] LNetError: 84366:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.814058] LNet: Removed LNI 192.168.204.144@tcp [ 2493.760191] Key type .llcrypt unregistered [ 2493.762773] Key type ._llcrypt unregistered [ 2517.561977] Key type ._llcrypt registered [ 2517.566572] Key type .llcrypt registered [ 2517.671612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2518.776202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.964983] alg: No test for adler32 (adler32-zlib) [ 2520.040788] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2520.265727] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2521.920261] Key type lgssc registered [ 2523.029804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2530.764404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2530.778370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.313640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2543.894158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.477175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.719469] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2549.730894] LustreError: 86468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.754332] LustreError: 86468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2554.854195] LustreError: 86471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.871679] LustreError: 86471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2557.024865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.272860] LustreError: 86468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.295114] LustreError: 86468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2557.306195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.531658] LustreError: 86472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.544873] LustreError: 86472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2563.004571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2567.652123] LustreError: 86471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.681972] LustreError: 86471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2571.434193] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.444677] Lustre: MGS: Client 1adc00d7-7ffd-4bd5-89e6-6c54511c4e60 (at 0@lo) reconnecting [ 2571.472988] LustreError: 86482:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2571.745429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2571.838940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2577.633574] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2592.226330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.241435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.243483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.264285] Lustre: Skipped 1 previous similar message [ 2597.347483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.367670] Lustre: Skipped 1 previous similar message [ 2598.435563] Lustre: server umount lustre-OST0000 complete [ 2602.466595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.471192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.478459] Lustre: Skipped 1 previous similar message [ 2602.486904] Lustre: Skipped 2 previous similar messages [ 2607.587318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.606487] Lustre: Skipped 1 previous similar message [ 2608.519964] Lustre: server umount lustre-MDT0000 complete [ 2612.121827] Lustre: server umount lustre-MDT0001 complete [ 2615.587726] Lustre: server umount MGS complete [ 2626.069600] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:09:11 (1787386151) [ 2651.661155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2660.604670] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2697.229714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.814472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2713.197159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2713.402815] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2713.443296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2713.521453] Lustre: lustre-MDT0001: new disk, initializing [ 2713.620858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2713.667305] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2713.684126] Lustre: 93656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2713.694908] LustreError: 93656:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2718.240525] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2728.493513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2728.812614] Lustre: lustre-OST0000: new disk, initializing [ 2728.814984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2728.820923] Lustre: 94422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2728.882216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.179086] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2738.786546] Lustre: 94878:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000035fe9593 x1874210111659520/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787386281 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2745.531915] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2752.581181] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.595264] Lustre: MGS: Client 61899482-e376-4755-88aa-9ade3c417c4a (at 0@lo) reconnecting [ 2752.659694] Lustre: 95467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2752.672212] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2752.690967] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2752.727655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2752.841095] Lustre: lustre-MDT0000: new disk, initializing [ 2753.063270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.099624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2753.136086] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2754.211460] Lustre: 95491:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000dfa747d5 x1874210111666688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787386297 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2757.975901] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2759.330388] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2763.318178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2763.331865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2763.418836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2789.345890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2789.358935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2789.374108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2791.912823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2791.927370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.736751] Lustre: server umount lustre-OST0000 complete [ 2797.538479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.542552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2797.569789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2797.588052] Lustre: Skipped 4 previous similar messages [ 2803.647939] Lustre: server umount lustre-MDT0000 complete [ 2814.372269] Lustre: server umount lustre-MDT0001 complete [ 2818.867086] Lustre: server umount MGS complete [ 2830.725438] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:12:36 (1787386356) [ 2848.722420] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2851.791452] Key type lgssc unregistered [ 2852.199228] LNet: 97987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2852.216241] LNetError: 97987:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2852.239215] LNet: Removed LNI 192.168.204.144@tcp [ 2852.921382] Key type .llcrypt unregistered [ 2852.924774] Key type ._llcrypt unregistered [ 2868.646431] Key type ._llcrypt registered [ 2868.649471] Key type .llcrypt registered [ 2868.789749] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2869.743029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2869.889462] alg: No test for adler32 (adler32-zlib) [ 2871.021690] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 2871.418590] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2873.160154] Key type lgssc registered [ 2874.400406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2883.016867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2883.043653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.566439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.415104] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2894.828611] LustreError: 99207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.864452] LustreError: 99207:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2899.808181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.951793] LustreError: 99206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.540663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2905.834437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2913.145195] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2918.327824] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2924.849525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.963911] LustreError: 101027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.202543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2929.333736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2931.133786] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2937.789541] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2944.994293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2944.999762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.002747] LustreError: Skipped 1 previous similar message [ 2945.005182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2945.010710] Lustre: Skipped 1 previous similar message [ 2945.019730] Lustre: Skipped 1 previous similar message [ 2950.127241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.141268] Lustre: Skipped 1 previous similar message [ 2955.247780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.259229] Lustre: Skipped 1 previous similar message [ 2959.328528] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2959.517627] Lustre: server umount lustre-OST0000 complete [ 2965.480133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.488244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.501043] Lustre: Skipped 1 previous similar message [ 2965.528135] Lustre: Skipped 1 previous similar message [ 2969.620760] Lustre: server umount lustre-MDT0000 complete [ 2970.593459] LustreError: 99201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.624751] LustreError: 99201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2973.155161] LustreError: 99189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386500 with bad export cookie 15266245858367457137 [ 2973.159687] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.167190] LustreError: 99189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2973.570700] Lustre: server umount lustre-MDT0001 complete [ 2984.070787] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2986.726770] Key type lgssc unregistered [ 2987.070664] LNet: 102742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2987.082472] LNetError: 102742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2987.100694] LNet: Removed LNI 192.168.204.144@tcp [ 2988.000923] Key type .llcrypt unregistered [ 2988.003091] Key type ._llcrypt unregistered [ 3005.385509] Key type ._llcrypt registered [ 3005.387822] Key type .llcrypt registered [ 3005.559348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3006.721909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3006.763863] alg: No test for adler32 (adler32-zlib) [ 3008.065666] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3008.468673] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3010.144272] Key type lgssc registered [ 3011.852277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3018.723821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3018.731260] LustreError: 103821:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3018.804847] LustreError: 103828:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3018.817373] LustreError: 103821:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4924 [ 3018.835462] LustreError: 103821:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3018.844148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.854846] LustreError: 103821:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3020.608225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.279938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.586179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3032.053274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3036.765447] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lsmod [ 3039.881514] LustreError: 105032:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3039.907174] LustreError: 105034:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3039.918047] LustreError: 105032:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4965 [ 3039.926354] LustreError: 105032:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3039.934699] LustreError: 105032:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3039.945607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.443059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.809854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3043.829181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3043.858990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.862563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.871037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.880151] Lustre: Skipped 1 previous similar message [ 3049.994563] Lustre: server umount lustre-MDT0000 complete [ 3050.983199] LustreError: 103860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.020547] LustreError: 103860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3053.034308] LustreError: 103865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.069940] LustreError: 103865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3053.794527] LustreError: 105072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386580 with bad export cookie 5564356385662044092 [ 3053.796957] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.815176] LustreError: 105072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3054.220676] Lustre: server umount lustre-MDT0001 complete [ 3068.993115] Lustre: server umount lustre-OST0000 complete [ 3078.019407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.583556] LustreError: 106098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.657774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.409493] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3083.749848] LustreError: 106099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.870375] LustreError: 106098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.406588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.906370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3097.941241] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3106.144662] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3111.772185] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3118.332977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.659264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.772524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3125.296126] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3133.968514] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3147.240271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3147.241199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.249786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3147.263063] Lustre: Skipped 1 previous similar message [ 3147.294960] Lustre: Skipped 3 previous similar messages [ 3152.374628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3152.378492] Lustre: Skipped 1 previous similar message [ 3153.264063] Lustre: server umount lustre-OST0000 complete [ 3157.474437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.491846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.497952] Lustre: Skipped 1 previous similar message [ 3167.716470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3167.723731] Lustre: Skipped 2 previous similar messages [ 3171.296303] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3171.543721] Lustre: server umount lustre-MDT0000 complete [ 3172.843571] LustreError: 106093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.870902] LustreError: 106093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3175.323908] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386702 with bad export cookie 5564356385662044792 [ 3175.331269] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.333467] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3175.750460] Lustre: server umount lustre-MDT0001 complete [ 3187.830669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3191.325110] Key type lgssc unregistered [ 3191.582841] LNet: 109654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3191.599816] LNetError: 109654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3191.617204] LNet: Removed LNI 192.168.204.144@tcp [ 3192.632341] Key type .llcrypt unregistered [ 3192.635604] Key type ._llcrypt unregistered [ 3218.994366] Key type ._llcrypt registered [ 3218.997096] Key type .llcrypt registered [ 3219.122546] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3220.348193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3220.368254] alg: No test for adler32 (adler32-zlib) [ 3221.567192] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3221.780752] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3223.472234] Key type lgssc registered [ 3224.904502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.343384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.371891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.904556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.372990] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3244.005200] LustreError: 111071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.021306] LustreError: 111071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3247.016940] LustreError: 111070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.462374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.903831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3253.862024] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3262.287635] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3268.883552] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.441866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.972742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3279.022778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3283.318635] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3291.432288] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3297.712208] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3302.703875] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3309.030331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.034349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3309.054108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3313.934957] Lustre: server umount lustre-OST0000 complete [ 3319.265849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.271539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3319.287569] Lustre: Skipped 2 previous similar messages [ 3319.313824] Lustre: Skipped 2 previous similar messages [ 3324.386417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.402982] Lustre: Skipped 1 previous similar message [ 3329.507285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.517824] Lustre: Skipped 1 previous similar message [ 3332.064499] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3332.200617] Lustre: server umount lustre-MDT0000 complete [ 3334.626632] LustreError: 113020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3334.670689] LustreError: 113020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3336.282151] LustreError: 113605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787386863 with bad export cookie 1302267624721196271 [ 3336.283692] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.294518] LustreError: 113605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3336.955543] Lustre: server umount lustre-MDT0001 complete [ 3349.813912] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3353.048630] Key type lgssc unregistered [ 3353.427588] LNet: 114713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3353.438564] LNetError: 114713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.464219] LNet: Removed LNI 192.168.204.144@tcp [ 3354.458444] Key type .llcrypt unregistered [ 3354.459874] Key type ._llcrypt unregistered [ 3375.801371] Key type ._llcrypt registered [ 3375.803961] Key type .llcrypt registered [ 3375.935459] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3377.596807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3377.648769] alg: No test for adler32 (adler32-zlib) [ 3378.719779] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3379.109180] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3380.904177] Key type lgssc registered [ 3382.390108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3391.218408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3391.247281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.689774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3397.702532] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3402.220247] LustreError: 115935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.243850] LustreError: 115935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3407.333947] LustreError: 115934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.594514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.156406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3413.149915] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3420.970818] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3427.073950] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3433.870658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.208633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3437.312715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3440.949385] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3449.251533] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3462.121585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3462.131211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.157882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3464.165739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.168094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3468.206343] Lustre: server umount lustre-OST0000 complete [ 3474.401869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.423179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3474.437425] Lustre: Skipped 2 previous similar messages [ 3479.530197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3479.535358] Lustre: Skipped 1 previous similar message [ 3484.644115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.657243] Lustre: Skipped 1 previous similar message [ 3486.176132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3486.335838] Lustre: server umount lustre-MDT0000 complete [ 3489.764555] LustreError: 115931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3489.799044] LustreError: 115931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3490.165051] LustreError: 115916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387017 with bad export cookie 1940417232909305024 [ 3490.170889] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.180872] LustreError: 115916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3490.520727] Lustre: server umount lustre-MDT0001 complete [ 3503.386255] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3506.307624] Key type lgssc unregistered [ 3506.583768] LNet: 119491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3506.592332] LNetError: 119491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3506.620568] LNet: Removed LNI 192.168.204.144@tcp [ 3507.565656] Key type .llcrypt unregistered [ 3507.568801] Key type ._llcrypt unregistered [ 3533.143276] Key type ._llcrypt registered [ 3533.144689] Key type .llcrypt registered [ 3533.273827] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3534.420583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3534.498738] alg: No test for adler32 (adler32-zlib) [ 3535.727628] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3536.017186] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3537.728277] Key type lgssc registered [ 3538.980847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3547.127048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3547.163576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.541535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3553.822949] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3558.888293] LustreError: 120908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.913605] LustreError: 120908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3562.443551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.015425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.592512] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3575.307376] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3581.841205] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3589.570483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.035367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3593.148563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3597.291414] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3606.324299] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3612.735826] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3616.575260] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3616.581153] Lustre: Skipped 2 previous similar messages [ 3620.983741] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3621.030911] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3621.485788] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3621.496411] Lustre: Skipped 2 previous similar messages [ 3626.632425] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3626.645495] Lustre: Skipped 1 previous similar message [ 3632.978550] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3632.984299] Lustre: Skipped 2 previous similar messages [ 3635.892086] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 3646.875855] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.44/24@tcp in log lustre-MDT0000 [ 3646.887399] Lustre: Skipped 2 previous similar messages [ 3653.767431] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 3653.771923] Lustre: Skipped 1 previous similar message [ 3660.800443] LustreError: 124902:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0xa6:0x0].0x0 (0000000058ce6103) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3660.806230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3660.847530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.864504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3661.339686] Lustre: server umount lustre-OST0000 complete [ 3668.136802] Lustre: server umount lustre-MDT0000 complete [ 3668.962454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.971851] LustreError: 124462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.983120] Lustre: Skipped 1 previous similar message [ 3669.035206] LustreError: 124462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3672.305471] LustreError: 123537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387199 with bad export cookie 5701307548898784396 [ 3672.313115] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.324098] LustreError: 123537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3683.102654] Lustre: server umount lustre-MDT0001 complete [ 3693.743161] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3698.056467] Key type lgssc unregistered [ 3698.470888] LNet: 125855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3698.475815] LNetError: 125855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3698.508555] LNet: Removed LNI 192.168.204.144@tcp [ 3699.875172] Key type .llcrypt unregistered [ 3699.876868] Key type ._llcrypt unregistered [ 3735.850920] Key type ._llcrypt registered [ 3735.854177] Key type .llcrypt registered [ 3735.995775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3737.412841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3737.446786] alg: No test for adler32 (adler32-zlib) [ 3738.576699] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3738.875569] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3740.616249] Key type lgssc registered [ 3742.081597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3750.464202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3750.502589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.060404] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3752.078849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.102958] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.44/24@tcp [ 3756.911216] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3762.660026] LustreError: 127420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.686920] LustreError: 127420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3765.563811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.929299] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3765.956936] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 3766.013462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3770.639619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3778.404754] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3785.199698] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3792.359942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3792.850243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.989614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3800.139319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3807.320369] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3822.568183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.579646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.603349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.636774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.637953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.663589] Lustre: Skipped 1 previous similar message [ 3827.597319] Lustre: server umount lustre-OST0000 complete [ 3832.804291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3832.814662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.834890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.874811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3835.875938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.904616] Lustre: Skipped 1 previous similar message [ 3840.993938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.010812] Lustre: Skipped 1 previous similar message [ 3846.112810] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3846.663530] Lustre: server umount lustre-MDT0000 complete [ 3850.209761] LustreError: 129162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3850.247272] LustreError: 129162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3850.697220] LustreError: 127402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387377 with bad export cookie 15005110616517673022 [ 3850.707064] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3851.199469] Lustre: server umount lustre-MDT0001 complete [ 3864.294229] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3868.131486] Key type lgssc unregistered [ 3868.453184] LNet: 131011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3868.461054] LNetError: 131011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3868.495594] LNet: Removed LNI 192.168.204.144@tcp [ 3869.777366] Key type .llcrypt unregistered [ 3869.787490] Key type ._llcrypt unregistered [ 3899.539146] Key type ._llcrypt registered [ 3899.540960] Key type .llcrypt registered [ 3899.656492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3916.503613] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3917.852167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3917.879993] alg: No test for adler32 (adler32-zlib) [ 3919.031128] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 3919.329393] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3921.080195] Key type lgssc registered [ 3922.562659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3960.384314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3960.448402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3961.885556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3961.956873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3962.115033] Lustre: lustre-MDT0000: new disk, initializing [ 3962.427595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3962.499218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3967.150705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3981.453849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.617508] Lustre: 134413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3981.672410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3981.676370] Lustre: Skipped 1 previous similar message [ 3981.736757] Lustre: lustre-MDT0001: new disk, initializing [ 3981.805932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3981.861971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3981.871541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3990.031473] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4000.437894] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4005.838157] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4013.287308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4013.540619] Lustre: lustre-OST0000: new disk, initializing [ 4013.544888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4013.550800] Lustre: 136382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4013.722317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4021.119653] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4021.326889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4021.338064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4021.563756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4031.134582] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4036.129513] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4036.339561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4041.315833] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4041.576755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4047.339545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.342692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4047.369468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4047.373951] Lustre: Skipped 1 previous similar message [ 4049.938078] Lustre: server umount lustre-OST0000 complete [ 4057.575182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.595937] Lustre: Skipped 1 previous similar message [ 4057.615865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4057.629900] Lustre: Skipped 1 previous similar message [ 4060.657140] Lustre: server umount lustre-MDT0000 complete [ 4065.850399] LustreError: 134403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387592 with bad export cookie 299262891203106994 [ 4065.854675] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4065.867883] LustreError: 134403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4066.391752] Lustre: server umount lustre-MDT0001 complete [ 4076.218308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.769370] LustreError: 138609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.800562] LustreError: 138609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4076.866657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.145307] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.791834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4086.248980] LustreError: 138609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.362209] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.927210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.402222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4097.150544] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4105.487413] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4111.137428] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4118.211765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4118.468866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4125.079808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4133.189799] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4139.062268] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4139.236173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4144.440681] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4144.827560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4154.876722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4155.070704] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (not set up) [ 4155.087929] Lustre: lustre-OST0001: new disk, initializing [ 4155.094834] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4155.113324] Lustre: 141984:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4155.214544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.675094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4161.698907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4161.839804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4163.865536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4172.900993] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4173.176322] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4177.454711] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4177.721618] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4184.025193] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b5c47373000.ost_server_uuid 50 [ 4185.758976] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b5c47373000.ost_server_uuid in IDLE state after 0 sec [ 4197.866180] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.871903] LustreError: Skipped 1 previous similar message [ 4197.874087] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.878397] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4197.879650] Lustre: Skipped 2 previous similar messages [ 4201.065570] Lustre: server umount lustre-OST0001 complete [ 4202.982280] LustreError: 140433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.991470] LustreError: 140433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4208.110038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.128324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.133897] Lustre: Skipped 2 previous similar messages [ 4213.223181] LustreError: 140434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.255515] LustreError: 140434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4218.341165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4218.359859] Lustre: Skipped 2 previous similar messages [ 4220.384293] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.640267] Lustre: server umount lustre-OST0000 complete [ 4225.515167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4225.521505] LustreError: Skipped 1 previous similar message [ 4225.527787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.538890] Lustre: Skipped 1 previous similar message [ 4231.074710] Lustre: server umount lustre-MDT0000 complete [ 4233.711315] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.737265] LustreError: 138610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4235.072850] LustreError: 138591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387762 with bad export cookie 299262891203107876 [ 4235.077255] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.081389] LustreError: 138591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4235.385649] Lustre: server umount lustre-MDT0001 complete [ 4254.585456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4257.323511] Key type lgssc unregistered [ 4257.609927] LNet: 144991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4257.619161] LNetError: 144991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4258.666882] LNet: Removed LNI 192.168.204.144@tcp [ 4259.675835] Key type .llcrypt unregistered [ 4259.678291] Key type ._llcrypt unregistered [ 4301.389612] Key type ._llcrypt registered [ 4301.393632] Key type .llcrypt registered [ 4301.576114] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4321.380227] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4323.025682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4323.138233] alg: No test for adler32 (adler32-zlib) [ 4324.317802] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4324.687560] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4326.385716] Key type lgssc registered [ 4327.877986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4366.794227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4366.836974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.066314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4368.098593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4368.161953] Lustre: lustre-MDT0000: new disk, initializing [ 4368.226252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.246806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4372.767476] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4383.996402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.136530] Lustre: 149146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4384.202642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4384.210996] Lustre: Skipped 1 previous similar message [ 4384.211520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4384.290225] Lustre: lustre-MDT0001: new disk, initializing [ 4384.398806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4384.428782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4384.441783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4389.383622] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4398.715343] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4404.296483] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4410.470713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.688465] Lustre: lustre-OST0000: new disk, initializing [ 4410.696438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4410.709239] Lustre: 151116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4410.819456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.986862] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4417.119443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4417.135540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4417.276897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4426.916932] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4431.964109] Lustre: Failing over lustre-OST0000 [ 4432.173468] Lustre: server umount lustre-OST0000 complete [ 4448.226904] Lustre: 147374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387959/real 1787387959] req@000000009a9a0f97 x1874212003284224/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4448.269334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.253027] Lustre: 147373:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387960/real 1787387960] req@0000000012e7aff0 x1874212003284352/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787387976 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4449.278727] Lustre: 147373:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4451.032408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.446624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.467803] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4454.432134] Lustre: 147374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387965/real 1787387965] req@00000000639f23fb x1874212003285376/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4457.358641] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4458.612490] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4459.304520] Lustre: 147372:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387970/real 1787387970] req@00000000a8529899 x1874212003286656/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4459.354132] Lustre: 147372:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4460.382412] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4460.385247] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4461.946278] Lustre: Failing over lustre-MDT0000 [ 4462.287333] Lustre: server umount lustre-MDT0000 complete [ 4464.033091] Lustre: 147371:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387975/real 1787387975] req@00000000b10a67e4 x1874212003287680/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787387991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4464.065897] Lustre: 147371:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4464.105200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.144827] Lustre: Skipped 3 previous similar messages [ 4469.223138] LustreError: 149160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4469.255222] LustreError: 149160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4470.566506] LustreError: 149156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.342330] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.360976] LustreError: 149159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4479.467581] LustreError: 150231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.483057] LustreError: 150231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4480.481928] Lustre: 147374:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787387991/real 1787387991] req@000000008a569a74 x1874212003296768/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787388007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4480.515748] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4482.519468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4485.603918] LustreError: 149155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.614606] LustreError: 149155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4490.721775] LustreError: 147370:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000755317c0 x1874212003302912/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4491.028206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.079355] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4495.525970] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4496.160731] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4496.382809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4496.398119] Lustre: Skipped 1 previous similar message [ 4496.426684] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4496.490674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4506.605338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4506.614197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.633144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4506.650160] Lustre: Skipped 1 previous similar message [ 4506.669542] Lustre: Skipped 1 previous similar message [ 4509.539935] Lustre: server umount lustre-OST0000 complete [ 4516.843312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.843954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.886373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.892492] Lustre: Skipped 1 previous similar message [ 4519.551301] Lustre: server umount lustre-MDT0000 complete [ 4521.954462] LustreError: 152767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.981234] LustreError: 152767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4523.224442] LustreError: 149140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388050 with bad export cookie 6575143025053008128 [ 4523.226212] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4523.243620] LustreError: 149140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4523.639292] Lustre: server umount lustre-MDT0001 complete [ 4534.338813] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4537.096908] Key type lgssc unregistered [ 4537.349623] LNet: 154772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4537.360667] LNetError: 154772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4537.377525] LNet: Removed LNI 192.168.204.144@tcp [ 4538.242278] Key type .llcrypt unregistered [ 4538.250451] Key type ._llcrypt unregistered [ 4574.666392] Key type ._llcrypt registered [ 4574.668404] Key type .llcrypt registered [ 4574.828421] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4576.249648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.334053] alg: No test for adler32 (adler32-zlib) [ 4577.577286] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 4577.938267] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4579.696296] Key type lgssc registered [ 4580.818850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4588.289717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4588.303764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.806558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.548964] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4599.278854] LustreError: 156294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.309488] LustreError: 156294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4603.744344] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4604.388383] LustreError: 156295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.557593] LustreError: 156324:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4607.570836] LustreError: 156324:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4607.578835] LustreError: 156324:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4607.896389] Lustre: server umount lustre-MDT0000 complete [ 4623.170769] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4633.616058] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4668.254314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.581515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4668.624222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4668.702712] Lustre: lustre-MDT0000: new disk, initializing [ 4668.809369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.837817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4674.408503] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4685.534620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.760392] Lustre: 159629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4685.912788] LustreError: 159640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.948029] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4685.955415] Lustre: Skipped 1 previous similar message [ 4686.091587] Lustre: lustre-MDT0001: new disk, initializing [ 4686.216776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4686.249530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4686.260284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4691.055536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4700.804448] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4707.097829] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4714.636246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4714.942486] Lustre: lustre-OST0000: new disk, initializing [ 4714.947507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4714.957567] Lustre: 161602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4715.066423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.960676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4721.750349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4721.778099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4721.864852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4729.506824] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4733.081741] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4734.595904] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:44:20 (1787388260) [ 4970.977382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4970.984705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4970.994231] Lustre: Skipped 2 previous similar messages [ 4971.008459] Lustre: Skipped 2 previous similar messages [ 4974.664835] Lustre: server umount lustre-MDT0000 complete [ 4976.103540] LustreError: 160913:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.122802] LustreError: 160913:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4978.455623] LustreError: 160441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388505 with bad export cookie 6678444361600536251 [ 4978.458796] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4978.466982] LustreError: 160441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4981.220357] LustreError: 159636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.220371] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.221085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4983.268295] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4984.919810] Lustre: server umount lustre-MDT0001 complete [ 5003.234221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5003.419524] Lustre: server umount lustre-OST0000 complete [ 5014.041293] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5024.177439] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5061.354535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.634202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5061.678872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5061.760284] Lustre: lustre-MDT0000: new disk, initializing [ 5061.863472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.880469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5066.066102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5076.608092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.719773] Lustre: 166263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5076.791435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5076.801754] Lustre: Skipped 1 previous similar message [ 5076.898608] Lustre: lustre-MDT0001: new disk, initializing [ 5076.988687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5077.040973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5077.060785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5082.078981] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5091.452485] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5097.242850] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5104.310572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5104.623129] Lustre: lustre-OST0000: new disk, initializing [ 5104.631867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5104.642294] Lustre: 168232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5104.734650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.336212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5110.354601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5110.475040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5111.120806] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5120.202408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5124.946285] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5125.184580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5129.813233] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5129.985331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5136.371725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.373238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.409215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5138.272932] Lustre: server umount lustre-OST0000 complete [ 5143.521653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5143.527049] LustreError: Skipped 1 previous similar message [ 5143.532237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5143.546776] Lustre: Skipped 1 previous similar message [ 5143.552611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5143.558706] Lustre: Skipped 1 previous similar message [ 5151.716265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5151.723171] Lustre: Skipped 2 previous similar messages [ 5157.348102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5157.646642] Lustre: server umount lustre-MDT0000 complete [ 5161.737884] LustreError: 167057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388688 with bad export cookie 6678444361600695081 [ 5161.741184] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.748143] LustreError: 167057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5162.161844] Lustre: server umount lustre-MDT0001 complete [ 5171.322784] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:51:36 (1787388696) [ 5187.856214] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5198.634529] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5229.504761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.699839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5229.729778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5229.812592] Lustre: lustre-MDT0000: new disk, initializing [ 5229.886137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.921458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5233.894437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5243.072058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.185669] Lustre: 172988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5243.482629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5247.890926] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5257.460319] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5263.283379] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5271.859644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.206970] Lustre: lustre-OST0000: new disk, initializing [ 5272.209989] Lustre: Skipped 1 previous similar message [ 5272.215330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5272.220605] Lustre: Skipped 2 previous similar messages [ 5272.225344] Lustre: 174967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5278.786060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5278.796455] Lustre: Skipped 1 previous similar message [ 5278.811975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5278.956175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5279.241355] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5289.341975] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5299.682534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5299.698462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.717232] Lustre: Skipped 1 previous similar message [ 5299.726203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5299.733683] Lustre: Skipped 3 previous similar messages [ 5303.088896] Lustre: server umount lustre-OST0000 complete [ 5309.922649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5309.925501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5309.942420] Lustre: Skipped 1 previous similar message [ 5309.953819] LustreError: Skipped 1 previous similar message [ 5313.030937] Lustre: server umount lustre-MDT0000 complete [ 5315.043331] LustreError: 174161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.067135] LustreError: 174161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5316.627436] LustreError: 173797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388843 with bad export cookie 6678444361600695963 [ 5316.630242] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5316.654186] LustreError: 173797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5317.153567] Lustre: server umount lustre-MDT0001 complete [ 5324.467188] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:54:10 (1787388850) [ 5342.961709] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5353.479277] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5388.170569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.494428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5388.527844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5388.613344] Lustre: lustre-MDT0000: new disk, initializing [ 5388.738918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.748925] Lustre: Skipped 2 previous similar messages [ 5388.764692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5393.415102] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5403.493624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.617744] Lustre: 179291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5403.972938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5408.350238] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5418.107105] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5424.390767] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5432.371864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5432.677626] Lustre: 181270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5438.542333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5438.637302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5440.590459] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5452.971824] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5464.548894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5464.552405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.558698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5464.572632] Lustre: Skipped 2 previous similar messages [ 5464.582961] Lustre: Skipped 2 previous similar messages [ 5466.780494] Lustre: server umount lustre-OST0000 complete [ 5477.398795] Lustre: server umount lustre-MDT0000 complete [ 5479.908134] LustreError: 180408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.920157] LustreError: 180408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5481.810141] LustreError: 179282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389008 with bad export cookie 6678444361600697048 [ 5481.820427] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.828239] LustreError: 179282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5482.472714] Lustre: server umount lustre-MDT0001 complete [ 5496.341147] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5500.274482] Key type lgssc unregistered [ 5500.553857] LNet: 183127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5500.563656] LNetError: 183127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5500.591050] LNet: Removed LNI 192.168.204.144@tcp [ 5501.520985] Key type .llcrypt unregistered [ 5501.523873] Key type ._llcrypt unregistered [ 5526.862606] Key type ._llcrypt registered [ 5526.868588] Key type .llcrypt registered [ 5527.038612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5528.221570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5528.273968] alg: No test for adler32 (adler32-zlib) [ 5529.326258] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 5529.532269] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5531.248211] Key type lgssc registered [ 5532.656491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5540.918572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5540.981781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.464289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.054692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5552.626625] LustreError: 184543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.640931] LustreError: 184543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5555.601964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.912835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5560.191653] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5567.301392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.491863] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5578.730594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5579.097383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.584651] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5587.319720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5592.684809] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5613.030163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.032794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5613.058268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5617.741502] Lustre: server umount lustre-OST0000 complete [ 5622.753446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5622.774749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.794984] Lustre: Skipped 1 previous similar message [ 5622.809249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5622.816790] Lustre: Skipped 1 previous similar message [ 5627.651798] Lustre: server umount lustre-MDT0000 complete [ 5628.385304] LustreError: 186289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.419817] LustreError: 186289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5631.559553] LustreError: 184525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389158 with bad export cookie 12626953504906340391 [ 5631.562781] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.579210] LustreError: 184525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5631.989803] Lustre: server umount lustre-MDT0001 complete [ 5642.375616] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5645.565438] Key type lgssc unregistered [ 5646.128151] LNet: 188099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5646.136582] LNetError: 188099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5646.154880] LNet: Removed LNI 192.168.204.144@tcp [ 5647.198993] Key type .llcrypt unregistered [ 5647.207679] Key type ._llcrypt unregistered [ 5674.429332] Key type ._llcrypt registered [ 5674.431737] Key type .llcrypt registered [ 5674.564714] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5676.105552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5676.139595] alg: No test for adler32 (adler32-zlib) [ 5677.301663] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 5677.524098] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5679.225240] Key type lgssc registered [ 5680.363644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.409415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5689.421502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.011116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5695.854548] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5701.604950] LustreError: 189517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.645916] LustreError: 189517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5705.146991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.612317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5710.497155] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5718.548203] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5723.806571] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5730.343285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.624965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.820385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5737.031941] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5744.112466] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.094603] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5752.303431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.318710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.319038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5756.207777] Lustre: server umount lustre-OST0000 complete [ 5764.545523] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b5c43211800.ost_server_uuid 50 [ 5766.257694] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5782.499761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5782.512638] LustreError: Skipped 1 previous similar message [ 5782.525307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.554562] Lustre: Skipped 1 previous similar message [ 5782.572819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5782.578802] Lustre: Skipped 1 previous similar message [ 5786.862114] Lustre: server umount lustre-MDT0000 complete [ 5788.134958] LustreError: 189511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.150154] LustreError: 189511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5789.936889] LustreError: 189497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389317 with bad export cookie 2150368741643394620 [ 5789.942778] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.950126] LustreError: 189497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5790.310284] Lustre: server umount lustre-MDT0001 complete [ 5798.169806] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 05:02:03 (1787389323) [ 5804.494089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.106703] LustreError: 193442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.200911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.457588] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5810.668087] LustreError: 193443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5815.785353] LustreError: 193442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.818055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.327393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5823.594612] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5831.067929] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5836.399461] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5842.921343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5843.254379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.192418] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5850.423904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5856.611915] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5863.100582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.334936] Lustre: lustre-OST0001: new disk, initializing [ 5863.340381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5863.346403] Lustre: 196282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5863.445264] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.665912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5869.674096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5869.761252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5869.884754] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5879.500100] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5885.237035] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5885.244646] Lustre: Skipped 2 previous similar messages [ 5885.418549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5885.433694] Lustre: Skipped 1 previous similar message [ 5885.442906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.340077] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5890.348843] Lustre: Skipped 2 previous similar messages [ 5896.161239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5896.305852] Lustre: server umount lustre-OST0000 complete [ 5900.587959] LustreError: 196289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.253905] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5902.502821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5906.455995] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5906.670578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5921.253444] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5921.261486] LustreError: 195264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5921.263262] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.263270] Lustre: Skipped 1 previous similar message [ 5921.268556] LustreError: Skipped 1 previous similar message [ 5921.271789] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5921.271799] Lustre: Skipped 5 previous similar messages [ 5921.291178] LustreError: 195264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5926.796979] Lustre: server umount lustre-OST0001 complete [ 5931.490265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.509777] Lustre: Skipped 2 previous similar messages [ 5941.731259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5941.739485] Lustre: Skipped 8 previous similar messages [ 5945.314347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5945.466511] Lustre: server umount lustre-MDT0000 complete [ 5949.201293] LustreError: 193424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389476 with bad export cookie 2150368741643395684 [ 5949.229130] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.239648] LustreError: 193424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5949.781776] Lustre: server umount lustre-MDT0001 complete [ 5978.265957] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 05:05:03 (1787389503) [ 5985.502687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5985.892602] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5985.929316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5986.153380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.108342] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5999.187312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.322214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5999.371122] Lustre: 199727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6004.105227] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6011.364849] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6017.196899] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6024.512140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.784937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6030.979360] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6032.017042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6038.268967] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6045.401346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6045.478482] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6052.171863] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6059.373467] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6062.562266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6062.578158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.614936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6068.538268] Lustre: server umount lustre-OST0000 complete [ 6070.048970] LustreError: 202013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.081339] LustreError: 202013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6088.674844] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.696383] Lustre: Skipped 2 previous similar messages [ 6091.436690] Lustre: server umount lustre-OST0001 complete [ 6096.881202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6096.908352] LustreError: Skipped 1 previous similar message [ 6101.635638] Lustre: server umount lustre-MDT0000 complete [ 6105.262322] LustreError: 199718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389632 with bad export cookie 2150368741643397098 [ 6105.266702] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6105.271305] LustreError: 199718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6105.587165] Lustre: server umount lustre-MDT0001 complete [ 6127.610646] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 05:07:33 (1787389653) [ 6144.052196] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6154.001911] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6189.896289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.141660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6190.190996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6190.287836] Lustre: lustre-MDT0000: new disk, initializing [ 6190.346533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.348932] Lustre: Skipped 3 previous similar messages [ 6190.360941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6194.764324] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6205.334718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.497663] Lustre: 207820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6205.536925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6205.539976] Lustre: Skipped 1 previous similar message [ 6205.590932] Lustre: lustre-MDT0001: new disk, initializing [ 6205.717756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6205.738336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6210.571879] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6220.139414] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.833387] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6232.981718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6233.268322] Lustre: lustre-OST0000: new disk, initializing [ 6233.275493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6233.282595] Lustre: 209790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6239.411040] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6239.822419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6239.831321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6240.023227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6249.399463] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6254.639404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6254.995693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6260.152264] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6260.425945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6265.827961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6265.835143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6265.838780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6265.838789] Lustre: Skipped 10 previous similar messages [ 6265.853098] Lustre: Skipped 3 previous similar messages [ 6268.770815] Lustre: server umount lustre-OST0000 complete [ 6279.101623] Lustre: server umount lustre-MDT0000 complete [ 6281.185883] LustreError: 207831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6281.201708] LustreError: 207831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 6282.768645] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389809 with bad export cookie 2150368741643398659 [ 6282.773523] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.799230] LustreError: 207813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6283.069603] Lustre: server umount lustre-MDT0001 complete [ 6292.232185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.374692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6306.086450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.097917] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6319.965100] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6325.704740] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6332.356456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6332.737073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.742301] Lustre: Skipped 4 previous similar messages [ 6338.709973] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6345.374559] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6349.263092] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6349.455564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6354.522161] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6354.863075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6358.500774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6358.507872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.531207] Lustre: Skipped 2 previous similar messages [ 6371.296392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6371.599947] Lustre: server umount lustre-OST0000 complete [ 6377.221159] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6380.987143] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6411.591975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6415.983984] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.020675] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6419.150913] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6421.165277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.190745] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6425.797784] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6428.375265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6436.840022] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6436.845029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6436.860578] LustreError: Skipped 2 previous similar messages [ 6436.879134] Lustre: Skipped 9 previous similar messages [ 6446.048201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6446.177494] Lustre: server umount lustre-OST0000 complete [ 6456.087249] Lustre: server umount lustre-MDT0000 complete [ 6457.316083] LustreError: 214134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6457.350233] LustreError: 214134:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6459.925820] LustreError: 211988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389987 with bad export cookie 2150368741643399541 [ 6459.930203] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.933627] LustreError: 211988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6460.220988] Lustre: server umount lustre-MDT0001 complete [ 6468.335280] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 05:13:14 (1787389994) [ 6473.908148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.425840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6486.034355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6490.664142] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6498.646298] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6505.567808] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6514.799166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6522.287679] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6530.227308] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6534.882282] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6535.096194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6539.463701] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6539.677964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6546.411920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.689787] Lustre: lustre-OST0001: new disk, initializing [ 6546.693270] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6546.699935] Lustre: 221346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6552.697726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6552.707411] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6552.824928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6553.094495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6562.931073] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6567.990893] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6568.217419] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6573.478800] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6573.716573] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6578.657845] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6578.658642] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.673421] LustreError: Skipped 2 previous similar messages [ 6578.697032] Lustre: Skipped 5 previous similar messages [ 6582.068908] Lustre: server umount lustre-OST0001 complete [ 6589.652928] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6589.978986] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6595.659580] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6595.898507] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6625.147493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.426249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.441212] Lustre: Skipped 5 previous similar messages [ 6630.909678] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6630.926158] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6630.933965] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6630.946730] Lustre: Skipped 1 previous similar message [ 6632.812784] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6641.263867] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6642.967924] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6651.024836] Lustre: server umount lustre-OST0001 complete [ 6673.773598] Lustre: server umount lustre-OST0000 complete [ 6683.656522] Lustre: server umount lustre-MDT0000 complete [ 6687.438441] LustreError: 217958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390214 with bad export cookie 2150368741643400507 [ 6687.440455] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6687.450284] LustreError: 217958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6693.894198] Lustre: server umount lustre-MDT0001 complete [ 6716.709840] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 05:17:22 (1787390242) [ 6723.318895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.639967] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6723.670080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6728.651328] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6737.248271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.347570] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6737.389627] Lustre: 226590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6742.195830] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6749.680338] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6756.528210] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6763.829352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.019208] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6769.309601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6775.883793] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6784.898183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6785.035253] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6790.448780] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6796.167996] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6797.586610] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6801.941507] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6802.127370] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6807.208624] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6807.451528] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6814.569193] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b5c474d1800.ost_server_uuid 50 [ 6816.468972] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b5c474d1800.ost_server_uuid in IDLE state after 0 sec [ 6817.453224] Lustre: Permanently deactivating lustre-OST0001 [ 6817.461648] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6820.105186] Lustre: Permanently reactivating lustre-OST0001 [ 6825.460475] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6825.466592] Lustre: Skipped 16 previous similar messages [ 6829.684082] Lustre: server umount lustre-OST0001 complete [ 6830.590564] LustreError: 230086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.599223] LustreError: 230086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6835.687142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.699783] Lustre: Skipped 9 previous similar messages [ 6840.168557] Lustre: server umount lustre-OST0000 complete [ 6845.412149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6845.428814] LustreError: Skipped 7 previous similar messages [ 6858.208196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6858.404200] Lustre: server umount lustre-MDT0000 complete [ 6862.277518] LustreError: 231003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390389 with bad export cookie 2150368741643401795 [ 6862.286100] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6862.294772] LustreError: 231003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6862.681624] Lustre: server umount lustre-MDT0001 complete [ 6887.619733] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 05:20:13 (1787390413) [ 6895.332806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.639632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6895.685978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6895.694909] Lustre: Skipped 5 previous similar messages [ 6900.871972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6909.696114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.825660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6909.859597] Lustre: 233289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6914.315843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6920.467704] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6925.555550] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6932.610051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6932.814204] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6932.843209] Lustre: Permanently deactivating lustre-OST0000 [ 6932.866957] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6938.140654] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6939.370861] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6946.886356] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6953.317692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6953.486515] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6961.052350] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6961.826946] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6969.544231] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6971.607964] Lustre: Permanently reactivating lustre-OST0000 [ 6971.644700] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6971.649265] Lustre: Skipped 2 previous similar messages [ 6980.696603] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6980.698612] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000d9689652, cur 1787390507 deadline 1787390495 last 1787390465 [ 6980.702197] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6980.703203] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6980.791541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6992.383845] Lustre: server umount lustre-OST0001 complete [ 7002.598276] Lustre: server umount lustre-OST0000 complete [ 7012.506879] Lustre: server umount lustre-MDT0000 complete [ 7016.718865] LustreError: 233281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390543 with bad export cookie 2150368741643403468 [ 7016.732932] LustreError: 233281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7016.733026] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.094907] Lustre: server umount lustre-MDT0001 complete [ 7028.964891] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7031.788337] Key type lgssc unregistered [ 7032.169873] LNet: 238170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7032.185701] LNetError: 238170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7032.204446] LNet: Removed LNI 192.168.204.144@tcp [ 7033.111497] Key type .llcrypt unregistered [ 7033.114957] Key type ._llcrypt unregistered [ 7057.818790] Key type ._llcrypt registered [ 7057.824807] Key type .llcrypt registered [ 7058.071084] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7059.610969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7059.915197] alg: No test for adler32 (adler32-zlib) [ 7061.173555] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7061.556296] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7063.376256] Key type lgssc registered [ 7064.803753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7075.575830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7075.593221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.245374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.621286] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7087.585636] LustreError: 239638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.626115] LustreError: 239638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7091.217693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.294845] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7091.320922] Lustre: 239624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.326820] Lustre: Permanently deactivating lustre-MDT0001 [ 7091.339836] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7091.639711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7097.752276] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7100.974786] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7106.187819] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7112.104646] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.859886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.154216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.221199] LustreError: 241661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7122.238580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7125.808698] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7127.541214] LustreError: 241661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.653196] LustreError: 241479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.253725] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7137.764951] LustreError: 241481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7137.791075] LustreError: 241481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7143.222981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.567762] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.054147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7149.070399] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7151.122841] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7159.711586] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7163.808125] Lustre: Permanently reactivating lustre-MDT0001 [ 7163.829203] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7163.841534] Lustre: Skipped 2 previous similar messages [ 7169.898078] Lustre: lustre-MDT0001: Client d4de8ced-1981-4b55-8d2b-bee7b0016524 (at 192.168.204.44@tcp) reconnecting [ 7170.790788] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.811367] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7170.821264] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7170.843991] LustreError: 243209:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7170.846694] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7170.865715] LustreError: 243209:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7170.887374] LustreError: 243209:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.144@tcp: cfg command failed: rc = -108 [ 7170.896461] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7170.896461] [ 7170.907317] LustreError: 239629:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7177.401129] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7179.545462] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7186.513115] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7186.753576] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7191.659813] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7191.896377] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7194.750732] Lustre: Permanently deactivating lustre-MDT0001 [ 7194.761986] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7194.764962] Lustre: Skipped 2 previous similar messages [ 7204.572043] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7210.469601] Lustre: server umount lustre-MDT0000 complete [ 7211.497460] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.505520] LustreError: 239633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7211.512427] Lustre: Skipped 1 previous similar message [ 7211.545658] LustreError: 239633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7214.901568] LustreError: 239619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390741 with bad export cookie 16813519452736832258 [ 7214.904670] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7214.909187] LustreError: 239619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7215.247548] Lustre: server umount lustre-MDT0001 complete [ 7219.749148] Lustre: server umount lustre-OST0000 complete [ 7232.914126] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 05:25:58 (1787390758) [ 7246.910380] Lustre: server umount lustre-OST0001 complete [ 7254.642715] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7265.733502] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7300.895569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.150786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7301.160713] Lustre: Skipped 2 previous similar messages [ 7301.188515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7301.278818] Lustre: lustre-MDT0000: new disk, initializing [ 7301.387552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.413671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7305.966331] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7316.875971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.038978] Lustre: 247900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7317.097246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7317.100787] Lustre: Skipped 1 previous similar message [ 7317.220440] Lustre: lustre-MDT0001: new disk, initializing [ 7317.361818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7317.433494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7317.450089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7321.987560] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7332.595408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7339.266560] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7347.340862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7347.672309] Lustre: lustre-OST0000: new disk, initializing [ 7347.682700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7347.691698] Lustre: 249872:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7347.794250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.410705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7353.418469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7353.543302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7354.535224] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7365.965254] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7369.060402] LustreError: 247909:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7377.873293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.001215] Lustre: lustre-OST0001: new disk, initializing [ 7378.004382] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7378.009758] Lustre: 250989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7379.071564] LustreError: 247909:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7379.096208] LustreError: 247907:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7383.679603] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7384.133834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7384.144838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7384.250401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7389.136356] LustreError: 247907:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7392.944513] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7394.667886] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7398.740428] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 7399.905294] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7399.920850] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.920860] Lustre: Skipped 2 previous similar messages [ 7399.926584] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7399.944199] LustreError: Skipped 1 previous similar message [ 7403.827673] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 7403.840333] Lustre: Skipped 1 previous similar message [ 7408.928010] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 7408.951270] Lustre: Skipped 2 previous similar messages [ 7411.168458] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7411.440775] Lustre: server umount lustre-OST0001 complete [ 7414.059374] LustreError: 249855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7430.626333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7430.641027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7430.672751] Lustre: Skipped 1 previous similar message [ 7430.682759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7430.692063] Lustre: Skipped 3 previous similar messages [ 7434.929339] Lustre: server umount lustre-OST0000 complete [ 7440.358917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7440.372355] LustreError: Skipped 1 previous similar message [ 7440.374348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.409950] Lustre: Skipped 1 previous similar message [ 7440.435580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7445.026971] Lustre: server umount lustre-MDT0000 complete [ 7449.238615] LustreError: 247893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390976 with bad export cookie 16813519452736834855 [ 7449.241284] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.253257] LustreError: 247893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7455.760763] Lustre: server umount lustre-MDT0001 complete [ 7468.488778] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7472.058781] Key type lgssc unregistered [ 7472.417389] LNet: 253022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7472.432267] LNetError: 253022:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7472.447408] LNet: Removed LNI 192.168.204.144@tcp [ 7473.821633] Key type .llcrypt unregistered [ 7473.827300] Key type ._llcrypt unregistered [ 7517.332442] Key type ._llcrypt registered [ 7517.337923] Key type .llcrypt registered [ 7517.446402] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7518.435336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7518.495467] alg: No test for adler32 (adler32-zlib) [ 7519.519578] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7519.764931] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7521.464270] Key type lgssc registered [ 7522.678267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7531.059498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7531.086724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.355973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7532.375268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7532.495743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.709513] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7545.576839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.751942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7545.824842] Lustre: 255247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7546.183179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7551.087089] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7559.753392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7566.153614] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7574.204291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.388329] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7574.579181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.319851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7580.650523] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7588.972334] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7642.081779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7642.082312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.092459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7642.119908] Lustre: Skipped 1 previous similar message [ 7647.201334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.220692] Lustre: Skipped 2 previous similar messages [ 7648.018517] Lustre: server umount lustre-OST0000 complete [ 7652.549607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7666.334853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7666.830426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.984435] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7671.994815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7671.996760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7672.023944] Lustre: Skipped 1 previous similar message [ 7674.345261] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7681.870693] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7723.498133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7723.499808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7723.502220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7723.509972] LustreError: Skipped 1 previous similar message [ 7723.537238] Lustre: Skipped 1 previous similar message [ 7726.341164] Lustre: server umount lustre-OST0000 complete [ 7730.657639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7730.669115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.693764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7730.700793] Lustre: Skipped 1 previous similar message [ 7733.736679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.114580] Lustre: server umount lustre-MDT0000 complete [ 7741.237451] LustreError: 255239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391268 with bad export cookie 17809952746681357993 [ 7741.238641] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7741.250261] LustreError: 255239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7741.642861] Lustre: server umount lustre-MDT0001 complete [ 7751.629352] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7754.474073] Key type lgssc unregistered [ 7754.841341] LNet: 260671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7754.852188] LNetError: 260671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7754.875637] LNet: Removed LNI 192.168.204.144@tcp [ 7756.009063] Key type .llcrypt unregistered [ 7756.016384] Key type ._llcrypt unregistered [ 7780.389258] Key type ._llcrypt registered [ 7780.393319] Key type .llcrypt registered [ 7780.479270] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7781.538431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7781.740147] alg: No test for adler32 (adler32-zlib) [ 7782.977849] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7783.362293] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7785.090266] Key type lgssc registered [ 7786.626338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7794.853173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7794.883606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.504949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7801.538117] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7805.927756] LustreError: 262087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7805.950751] LustreError: 262087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7810.391372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.692526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7815.377401] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7822.292669] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7827.509485] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7835.114358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.535201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.738896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7842.283140] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7850.378217] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7877.608154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.610127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.611464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.623185] LustreError: Skipped 1 previous similar message [ 7877.642063] Lustre: Skipped 1 previous similar message [ 7880.872379] Lustre: server umount lustre-OST0000 complete [ 7887.329151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7887.338895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7887.354664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7887.360436] Lustre: Skipped 1 previous similar message [ 7891.798345] Lustre: server umount lustre-MDT0000 complete [ 7892.961485] LustreError: 262084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7892.991428] LustreError: 262084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7896.263365] LustreError: 262067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391423 with bad export cookie 13524343748239680138 [ 7896.270298] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7896.799389] Lustre: server umount lustre-MDT0001 complete [ 7909.332761] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7912.894956] Key type lgssc unregistered [ 7913.301853] LNet: 266397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7913.317398] LNetError: 266397:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7913.361268] LNet: Removed LNI 192.168.204.144@tcp [ 7914.685185] Key type .llcrypt unregistered [ 7914.687241] Key type ._llcrypt unregistered [ 7938.032534] Key type ._llcrypt registered [ 7938.036275] Key type .llcrypt registered [ 7938.199705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7940.322769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7940.363786] alg: No test for adler32 (adler32-zlib) [ 7941.643065] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 7942.003704] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7943.738853] Key type lgssc registered [ 7945.246425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7954.492403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7954.531748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7956.152628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7961.433741] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7966.696184] LustreError: 267622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.737564] LustreError: 267622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7971.520108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.810204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7971.995681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7977.493892] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7985.688560] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7993.809823] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8002.367649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.653540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8009.537604] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8018.630333] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8026.240861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8041.960774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8041.961662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.969021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8041.979817] LustreError: Skipped 1 previous similar message [ 8042.008139] Lustre: Skipped 1 previous similar message [ 8047.093979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8047.107973] Lustre: Skipped 2 previous similar messages [ 8052.207840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.228075] Lustre: Skipped 1 previous similar message [ 8055.264220] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8055.604598] Lustre: server umount lustre-OST0000 complete [ 8062.441706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.472294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.482122] Lustre: Skipped 1 previous similar message [ 8072.676926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8072.683238] Lustre: Skipped 3 previous similar messages [ 8074.208975] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8074.510491] Lustre: server umount lustre-MDT0000 complete [ 8077.803812] LustreError: 267619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.826553] LustreError: 267619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8078.218614] LustreError: 267604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391605 with bad export cookie 16980786080428088547 [ 8078.237482] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.680977] Lustre: server umount lustre-MDT0001 complete [ 8089.955297] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8092.696458] Key type lgssc unregistered [ 8092.956719] LNet: 271326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.965627] LNetError: 271326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8093.995692] LNet: Removed LNI 192.168.204.144@tcp [ 8094.923106] Key type .llcrypt unregistered [ 8094.926408] Key type ._llcrypt unregistered [ 8121.334562] Key type ._llcrypt registered [ 8121.336864] Key type .llcrypt registered [ 8121.521782] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8122.553464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8122.577954] alg: No test for adler32 (adler32-zlib) [ 8123.820489] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8124.133236] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8125.913437] Key type lgssc registered [ 8127.423950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.237510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8136.269532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.737682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.936081] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8147.944354] LustreError: 272744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8147.988111] LustreError: 272744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8153.062272] LustreError: 272743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8153.852377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.245970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8159.677404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8168.912797] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8175.060835] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8182.299674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.820414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.869524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8189.151144] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8196.703333] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8224.739759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8224.754717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.773865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.251025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.258746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.296298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.303436] Lustre: Skipped 1 previous similar message [ 8227.141474] Lustre: server umount lustre-OST0000 complete [ 8230.373196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.377761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.399164] Lustre: Skipped 1 previous similar message [ 8235.498078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8235.514468] Lustre: Skipped 1 previous similar message [ 8237.115176] Lustre: server umount lustre-MDT0000 complete [ 8240.609347] LustreError: 272739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.629471] LustreError: 272739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8242.298655] LustreError: 272725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391769 with bad export cookie 12710853393411573289 [ 8242.313778] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8242.318120] LustreError: 272725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8242.692296] Lustre: server umount lustre-MDT0001 complete [ 8255.121344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8258.154514] Key type lgssc unregistered [ 8258.525749] LNet: 277048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8258.532607] LNetError: 277048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8258.551657] LNet: Removed LNI 192.168.204.144@tcp [ 8259.568712] Key type .llcrypt unregistered [ 8259.570627] Key type ._llcrypt unregistered [ 8277.655467] Key type ._llcrypt registered [ 8277.659189] Key type .llcrypt registered [ 8277.818525] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8279.137462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8279.197914] alg: No test for adler32 (adler32-zlib) [ 8280.282064] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8280.521502] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8282.187610] Key type lgssc registered [ 8283.602492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8292.194032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8292.229924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.840452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.039698] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8303.081176] LustreError: 278289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.101200] LustreError: 278289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8308.201416] LustreError: 278291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.455139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.755254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.589826] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8321.977526] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8328.163300] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8335.693340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8336.241870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.311459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8343.235773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8352.503184] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8370.154129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.155682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.187312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.204817] LustreError: Skipped 1 previous similar message [ 8374.104734] Lustre: server umount lustre-OST0000 complete [ 8380.386910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.387336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.387349] Lustre: Skipped 1 previous similar message [ 8380.392910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.465244] Lustre: Skipped 2 previous similar messages [ 8385.093939] Lustre: server umount lustre-MDT0000 complete [ 8385.509751] LustreError: 278269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.534198] LustreError: 278269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8389.298705] LustreError: 278254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391916 with bad export cookie 14084998745563539626 [ 8389.310942] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8389.314769] LustreError: 278254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8389.804975] Lustre: server umount lustre-MDT0001 complete [ 8405.018532] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8408.620161] Key type lgssc unregistered [ 8409.054697] LNet: 281993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8409.077655] LNetError: 281993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8409.094551] LNet: Removed LNI 192.168.204.144@tcp [ 8410.134412] Key type .llcrypt unregistered [ 8410.136704] Key type ._llcrypt unregistered [ 8424.407688] Key type ._llcrypt registered [ 8424.417464] Key type .llcrypt registered [ 8424.543891] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8444.108072] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8460.071788] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8461.614902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8461.638454] alg: No test for adler32 (adler32-zlib) [ 8462.999271] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8463.298352] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8465.088220] Key type lgssc registered [ 8466.537771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8503.281562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8503.346786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.606975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8504.641172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8504.735761] Lustre: lustre-MDT0000: new disk, initializing [ 8504.806737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8504.827722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8509.052743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8519.053740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.191927] Lustre: 285545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8519.258951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8519.262627] Lustre: Skipped 1 previous similar message [ 8519.328765] Lustre: lustre-MDT0001: new disk, initializing [ 8519.432626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.467767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8519.474592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8523.698178] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8532.828230] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8537.812380] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8543.968405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8544.277357] Lustre: lustre-OST0000: new disk, initializing [ 8544.282697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8544.290316] Lustre: 287514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8544.381314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8549.958618] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8553.537145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8553.558039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8553.677192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8558.056114] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8561.767577] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8561.963217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8566.281311] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8566.517673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8569.320027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8569.326284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8569.339303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8574.443419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8574.452369] Lustre: Skipped 1 previous similar message [ 8574.899588] Lustre: server umount lustre-OST0000 complete [ 8579.564027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8579.586503] Lustre: Skipped 1 previous similar message [ 8579.592789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8584.675162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8584.680720] Lustre: Skipped 2 previous similar messages [ 8584.932492] Lustre: server umount lustre-MDT0000 complete [ 8588.743431] LustreError: 285536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392115 with bad export cookie 16123973015120180622 [ 8588.755826] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8588.757883] LustreError: 285536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8589.065355] Lustre: server umount lustre-MDT0001 complete [ 8596.994646] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:48:42 (1787392122) [ 8603.462819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.052148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8608.657468] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8609.252728] LustreError: 289927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.273070] LustreError: 289927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8614.371912] LustreError: 289926:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8616.925353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.258886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.127659] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8629.056936] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8634.802437] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8641.889036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8642.316780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.643465] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8656.724154] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8670.181190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8670.191581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.204609] Lustre: Skipped 1 previous similar message [ 8670.211291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.255015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8673.255869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.307634] Lustre: server umount lustre-OST0000 complete [ 8683.499099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8683.505959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8683.517305] Lustre: Skipped 1 previous similar message [ 8683.541960] Lustre: Skipped 3 previous similar messages [ 8694.240537] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8694.527888] Lustre: server umount lustre-MDT0000 complete [ 8698.272116] LustreError: 289907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392225 with bad export cookie 16123973015120181504 [ 8698.275856] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8698.288061] LustreError: 289907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8698.583919] Lustre: server umount lustre-MDT0001 complete [ 8709.083209] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8712.158671] Key type lgssc unregistered [ 8712.566950] LNet: 293484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8712.574930] LNetError: 293484:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8713.643795] LNet: Removed LNI 192.168.204.144@tcp [ 8714.535901] Key type .llcrypt unregistered [ 8714.540262] Key type ._llcrypt unregistered [ 8741.010641] Key type ._llcrypt registered [ 8741.014443] Key type .llcrypt registered [ 8741.134370] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8756.597322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8757.537508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8757.601088] alg: No test for adler32 (adler32-zlib) [ 8758.718881] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 8758.957137] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8760.648277] Key type lgssc registered [ 8761.775644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8803.226667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8803.253180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.507078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.608318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8804.694300] Lustre: lustre-MDT0000: new disk, initializing [ 8804.801919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8804.830712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8809.689199] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8819.564926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.715373] Lustre: 297313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8819.765155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8819.770420] Lustre: Skipped 1 previous similar message [ 8819.860634] Lustre: lustre-MDT0001: new disk, initializing [ 8819.971586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8820.005404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8820.015816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8824.100568] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8833.774540] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8839.530433] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8846.871840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8847.155243] Lustre: lustre-OST2710: new disk, initializing [ 8847.159550] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8847.165221] Lustre: 299281:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8847.276581] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8853.585775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8855.080605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8855.088819] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8855.210391] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8862.535381] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8869.196787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.368909] Lustre: lustre-OST03e8: new disk, initializing [ 8869.375896] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8869.390680] Lustre: 300344:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8869.515293] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8875.603488] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8875.614038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8875.621416] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8875.731163] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8882.294410] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8889.289369] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8889.525667] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8894.851787] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8895.111732] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8900.221749] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8900.462859] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8905.534386] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8905.756760] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8911.858925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.863246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8911.871470] Lustre: Skipped 1 previous similar message [ 8912.463734] Lustre: server umount lustre-MDT0000 complete [ 8915.907459] LustreError: 300343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392442 with bad export cookie 4486618909647574661 [ 8915.921023] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.298462] Lustre: server umount lustre-MDT0001 complete [ 8929.189578] Lustre: server umount lustre-OST2710 complete [ 8933.217089] Lustre: 295113:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392444/real 1787392444] req@000000007f64f9e5 x1874216653751296/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787392460 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8933.274561] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.308470] Lustre: Skipped 2 previous similar messages [ 8933.447536] Lustre: server umount lustre-OST03e8 complete [ 8956.656807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8968.932259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9008.193514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.444349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9008.463207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9008.534279] Lustre: lustre-MDT0000: new disk, initializing [ 9008.599690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9008.614431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9013.056390] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9024.930122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9025.126504] Lustre: 305593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9025.225608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9025.231490] Lustre: Skipped 1 previous similar message [ 9025.299307] Lustre: lustre-MDT0001: new disk, initializing [ 9025.386561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9025.430051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9025.439334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9030.974256] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9040.444835] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9046.127117] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9052.682054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9052.989919] Lustre: lustre-OST0000: new disk, initializing [ 9053.007424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9053.016389] Lustre: 307562:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9053.115412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.381231] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9060.409650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9060.424096] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9060.510704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9069.491448] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9074.261906] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9074.594459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9079.569466] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9079.775667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9086.435691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9086.438889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.446874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9086.446883] Lustre: Skipped 3 previous similar messages [ 9086.452404] LustreError: Skipped 1 previous similar message [ 9086.500536] Lustre: Skipped 1 previous similar message [ 9087.878365] Lustre: server umount lustre-OST0000 complete [ 9092.067582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9092.076162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9092.089380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9092.093721] Lustre: Skipped 1 previous similar message [ 9096.684588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.685803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.707830] Lustre: Skipped 1 previous similar message [ 9097.904496] Lustre: server umount lustre-MDT0000 complete [ 9101.146846] LustreError: 305585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392628 with bad export cookie 4486618909647576341 [ 9101.148718] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9101.158382] LustreError: 305585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9101.448722] Lustre: server umount lustre-MDT0001 complete [ 9109.118359] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:57:14 (1787392634) [ 9110.382835] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9112.111649] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:57:17 (1787392637) [ 9115.259418] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9140.007129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9140.215703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9140.244845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9140.338387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9145.408697] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9154.767024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.831811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9154.859469] Lustre: 311422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9160.236219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9185.067566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9185.312225] Lustre: Denying initial registration attempt from nid 192.168.204.144@tcp, specified as failover [ 9185.323494] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9185.332593] LustreError: 312881:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9185.343236] LustreError: 312881:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9185.351782] LustreError: 312881:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9185.436417] Lustre: server umount lustre-OST0000 complete [ 9185.438394] LustreError: 312881:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9196.001430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9196.004093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.005651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9196.033706] Lustre: Skipped 1 previous similar message [ 9197.985168] Lustre: server umount lustre-MDT0000 complete [ 9201.150886] LustreError: 311434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9201.176814] LustreError: 311434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9201.472905] LustreError: 311414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392728 with bad export cookie 4486618909647577223 [ 9201.478098] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9201.486354] LustreError: 311414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9201.697648] Lustre: server umount lustre-MDT0001 complete [ 9209.598793] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9211.986026] Key type lgssc unregistered [ 9212.294531] LNet: 314001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9212.301278] LNetError: 314001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9212.316526] LNet: Removed LNI 192.168.204.144@tcp [ 9213.018449] Key type .llcrypt unregistered [ 9213.022570] Key type ._llcrypt unregistered [ 9222.457972] Key type ._llcrypt registered [ 9222.459858] Key type .llcrypt registered [ 9222.543264] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9223.245064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9223.276678] alg: No test for adler32 (adler32-zlib) [ 9224.303395] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9224.548619] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9226.216180] Key type lgssc registered [ 9227.227579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9269.745619] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9278.070610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9278.091437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.439791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9279.460259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9279.672132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.183259] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9291.836511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.936371] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9291.965855] Lustre: 316765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9292.249358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9296.858629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9304.410266] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9311.855923] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9331.891767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9332.079316] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9332.278601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.636802] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9344.421758] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9349.094948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9349.106667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.118732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9349.609210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.632954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9349.637444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9352.613844] Lustre: server umount lustre-OST0000 complete [ 9358.820100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9358.826196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.838146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9362.615586] Lustre: server umount lustre-MDT0000 complete [ 9366.320927] LustreError: 318639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392893 with bad export cookie 3424014611674456077 [ 9366.325658] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9366.330643] LustreError: 318639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9366.633123] Lustre: server umount lustre-MDT0001 complete [ 9376.752901] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9379.421923] Key type lgssc unregistered [ 9379.751775] LNet: 320346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9379.762834] LNetError: 320346:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9379.789254] LNet: Removed LNI 192.168.204.144@tcp [ 9380.628832] Key type .llcrypt unregistered [ 9380.631307] Key type ._llcrypt unregistered [ 9404.418576] Key type ._llcrypt registered [ 9404.420876] Key type .llcrypt registered [ 9404.520977] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9405.296459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9405.339762] alg: No test for adler32 (adler32-zlib) [ 9406.385577] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9406.648541] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9408.361194] Key type lgssc registered [ 9409.391501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9416.897252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9416.913820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9418.409862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.258392] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9428.003336] LustreError: 321766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.021876] LustreError: 321766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9431.119431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.730690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9436.821896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9445.419824] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9451.280707] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9459.773154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9460.150278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.807404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9474.648978] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9487.847600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9487.864510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9487.883770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.770852] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 9491.938929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.957765] Lustre: Skipped 1 previous similar message [ 9493.840325] Lustre: server umount lustre-MDT0000 complete [ 9496.862952] LustreError: 321763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.552601] LustreError: 321747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393024 with bad export cookie 13086035662013647760 [ 9497.565254] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.991973] LustreError: 321761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9502.022462] LustreError: 321761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9502.042864] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 9502.053296] Lustre: Skipped 3 previous similar messages [ 9502.181654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.001995] Lustre: server umount lustre-MDT0001 complete [ 9510.682883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9518.936241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.617749] LustreError: 321075:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000030e0e29a x1874217332157696/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9524.044796] LustreError: 323599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.060614] LustreError: 323599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9524.135669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9524.196642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9529.551883] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9532.262698] LustreError: 323600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9532.276665] LustreError: 323600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9532.280237] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9537.896831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.120924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9542.266218] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9543.494705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9551.043952] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9552.790992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9558.159537] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9559.764694] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9568.226624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.243828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.248057] Lustre: Skipped 1 previous similar message [ 9571.554065] Lustre: server umount lustre-OST0000 complete [ 9578.469441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.477087] Lustre: Skipped 1 previous similar message [ 9578.484846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9578.498057] Lustre: Skipped 1 previous similar message [ 9581.448741] Lustre: server umount lustre-MDT0000 complete [ 9583.587292] LustreError: 325382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.599819] LustreError: 325382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9584.904391] LustreError: 321748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393111 with bad export cookie 13086035662013655460 [ 9584.909508] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9584.919824] LustreError: 321748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9585.184687] Lustre: server umount lustre-MDT0001 complete [ 9594.852541] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9598.021477] Key type lgssc unregistered [ 9598.415610] LNet: 328053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9598.427304] LNetError: 328053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9598.441132] LNet: Removed LNI 192.168.204.144@tcp [ 9599.802541] Key type .llcrypt unregistered [ 9599.806207] Key type ._llcrypt unregistered [ 9625.105585] Key type ._llcrypt registered [ 9625.107620] Key type .llcrypt registered [ 9625.245745] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9626.212929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9626.292117] alg: No test for adler32 (adler32-zlib) [ 9627.387956] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9627.711978] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9629.416478] Key type lgssc registered [ 9630.546548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9639.235922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9639.250462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9640.659668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.817077] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9650.146984] LustreError: 329473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9650.159134] LustreError: 329473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9652.188115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.575705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9656.911292] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9664.493379] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9670.611919] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9675.236153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.249426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9675.255354] Lustre: Skipped 1 previous similar message [ 9675.275533] Lustre: Skipped 1 previous similar message [ 9677.350208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9677.354150] Lustre: Skipped 1 previous similar message [ 9679.884706] Lustre: server umount lustre-MDT0000 complete [ 9682.401361] LustreError: 330200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9682.878242] LustreError: 329455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393209 with bad export cookie 6588992159755017011 [ 9682.891478] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9682.897038] LustreError: 329455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9683.265890] Lustre: server umount lustre-MDT0001 complete [ 9698.126144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.407615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9698.443944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9698.697297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.329842] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9711.956326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.107223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9712.173694] Lustre: 332062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9712.551100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9716.716972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9724.614304] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9730.761702] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9741.177321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9741.488646] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9741.497612] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9741.507893] LustreError: 333947:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9741.513584] LustreError: 333947:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9741.524535] LustreError: 333947:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9741.628672] Lustre: server umount lustre-OST0000 complete [ 9741.632931] LustreError: 333947:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9751.530990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9751.722770] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9751.930932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.927874] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9761.448395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9764.959509] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9772.263855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9772.360484] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9772.425421] Lustre: lustre-OST0001: new disk, initializing [ 9772.433342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9772.442644] Lustre: 335488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9772.529764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9778.198252] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9781.915172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9781.927164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9782.047695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9785.889594] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9792.487290] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9792.496973] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9792.506482] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9792.509896] Lustre: Skipped 1 previous similar message [ 9792.519040] Lustre: Skipped 1 previous similar message [ 9794.854610] Lustre: server umount lustre-OST0001 complete [ 9797.610883] LustreError: 334414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.635612] LustreError: 334414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9802.210236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.233986] LustreError: Skipped 1 previous similar message [ 9802.244866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.275309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.728315] LustreError: 334414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.767042] LustreError: 334414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9805.087027] Lustre: server umount lustre-OST0000 complete [ 9809.891942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9809.908995] LustreError: Skipped 1 previous similar message [ 9809.912381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.939442] Lustre: Skipped 1 previous similar message [ 9809.955596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9809.964445] Lustre: Skipped 1 previous similar message [ 9815.550402] Lustre: server umount lustre-MDT0000 complete [ 9818.081146] LustreError: 333287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.601846] LustreError: 332052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393345 with bad export cookie 6588992159755017564 [ 9818.608395] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9818.616276] LustreError: 332052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9818.923567] Lustre: server umount lustre-MDT0001 complete [ 9829.313965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 9832.404138] Key type lgssc unregistered [ 9832.776689] LNet: 337445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9832.785677] LNetError: 337445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9832.806270] LNet: Removed LNI 192.168.204.144@tcp [ 9833.679821] Key type .llcrypt unregistered [ 9833.684944] Key type ._llcrypt unregistered [ 9908.227404] Key type ._llcrypt registered [ 9908.229640] Key type .llcrypt registered [ 9908.392654] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 9924.243112] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 9925.353585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9925.373390] alg: No test for adler32 (adler32-zlib) [ 9926.425992] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [ 9926.697351] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 9928.408337] Key type lgssc registered [ 9930.193328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9968.842575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9968.882683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9970.175786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9970.225148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9970.351785] Lustre: lustre-MDT0000: new disk, initializing [ 9970.496604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9970.521883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9975.302221] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 9986.647671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.827959] Lustre: 342378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9986.907687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9986.914733] Lustre: Skipped 1 previous similar message [ 9987.055384] Lustre: lustre-MDT0001: new disk, initializing [ 9987.163109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9987.198421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9987.210222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9992.038716] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10001.631860] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10007.241389] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10013.437963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.659327] Lustre: lustre-OST0000: new disk, initializing [10013.663448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10013.668071] Lustre: 344346:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10013.747036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10018.988935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10022.984717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10022.992839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10023.124287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10029.120822] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10033.086540] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10033.324742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10038.204672] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10038.385806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10043.873337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10043.878578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.888183] Lustre: Skipped 1 previous similar message [10043.889651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10046.792685] Lustre: server umount lustre-OST0000 complete [10053.606799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10053.615460] LustreError: Skipped 1 previous similar message [10053.625795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.648676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10053.655392] Lustre: Skipped 1 previous similar message [10056.759303] Lustre: server umount lustre-MDT0000 complete [10060.394113] LustreError: 342373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393587 with bad export cookie 2825965213527977436 [10060.409470] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10060.420982] LustreError: 342373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10060.845227] Lustre: server umount lustre-MDT0001 complete [10070.166959] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 06:13:15 (1787393595) [10079.243602] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 06:13:24 (1787393604) [10087.865348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10088.387863] LustreError: 347102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.416187] LustreError: 347102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10088.488299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.544052] LustreError: 347103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10093.723431] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10098.666928] LustreError: 347102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.852951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10103.284430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10107.893567] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10116.332641] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10122.571054] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10129.699194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10130.047192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.031738] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10144.891410] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10157.258676] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10159.083873] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10160.883731] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10162.769327] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10164.927417] Lustre: Failing over lustre-MDT0000 [10165.317449] Lustre: server umount lustre-MDT0000 complete [10168.136182] LustreError: 347099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.318676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10169.334287] Lustre: Skipped 3 previous similar messages [10173.216751] LustreError: 347097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.239884] LustreError: 347097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10183.462553] LustreError: 347099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.479903] LustreError: 347099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10184.650309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10184.894845] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.163889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10185.220551] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10189.601138] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10190.319920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10190.344406] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10190.356839] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10190.405925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10199.956410] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10201.583203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10205.495636] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10207.698948] Lustre: Failing over lustre-MDT0000 [10207.965677] Lustre: server umount lustre-MDT0000 complete [10210.123225] LustreError: 349361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.154549] LustreError: 349361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10210.786383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10210.788814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.813417] Lustre: Skipped 2 previous similar messages [10226.145115] Lustre: 340600:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787393737/real 1787393737] req@0000000031040c75 x1874217877463680/t0(0) o400->MGC192.168.204.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787393753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10226.180436] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10227.535255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.608791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10236.672336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10236.722300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10241.269142] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10242.044771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10242.054682] Lustre: Skipped 2 previous similar messages [10242.101020] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [10242.156717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10250.110578] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10251.536956] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10262.503764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10262.509246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.511500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10262.511507] Lustre: Skipped 2 previous similar messages [10262.567537] Lustre: Skipped 2 previous similar messages [10263.942775] Lustre: server umount lustre-MDT0000 complete [10267.620632] LustreError: 347097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10267.653772] LustreError: 347097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [10267.849218] LustreError: 352236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393794 with bad export cookie 2825965213527980145 [10267.855144] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10267.857342] LustreError: 352236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10268.317492] Lustre: server umount lustre-MDT0001 complete [10282.721678] Lustre: server umount lustre-OST0000 complete [10298.729813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.494884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10310.740818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.125636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10311.138625] Lustre: Skipped 1 previous similar message [10315.799139] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10323.227491] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10328.704951] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10336.124166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.717779] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10344.703537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10350.121678] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10358.245300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10358.257082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10358.277713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10358.280257] Lustre: Skipped 2 previous similar messages [10361.588222] Lustre: server umount lustre-OST0000 complete [10367.474604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10367.482277] LustreError: Skipped 1 previous similar message [10367.485733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10367.497339] Lustre: Skipped 2 previous similar messages [10371.915945] Lustre: server umount lustre-MDT0000 complete [10375.427067] LustreError: 353447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393902 with bad export cookie 2825965213527980894 [10375.443022] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10375.742631] Lustre: server umount lustre-MDT0001 complete [10385.618506] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10388.555987] Key type lgssc unregistered [10388.849658] LNet: 356998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10388.854967] LNetError: 356998:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10388.875698] LNet: Removed LNI 192.168.204.144@tcp [10389.746903] Key type .llcrypt unregistered [10389.749607] Key type ._llcrypt unregistered [10415.384846] Key type ._llcrypt registered [10415.390675] Key type .llcrypt registered [10415.472532] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10416.479743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10416.501870] alg: No test for adler32 (adler32-zlib) [10417.472191] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [10417.676367] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10419.320392] Key type lgssc registered [10420.517889] Lustre: Echo OBD driver; http://www.lustre.org/ [10428.207122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10428.227316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10429.771139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10434.303416] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10439.139612] LustreError: 358514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10439.162088] LustreError: 358514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10441.959063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10442.386920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10446.597288] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10453.312508] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10458.108764] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10464.441684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10464.859150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10470.655450] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10474.106276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10477.719755] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10488.806181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10488.822298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.851038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.827876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10489.838380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10489.857470] Lustre: Skipped 1 previous similar message [10490.000941] Lustre: server umount lustre-MDT0000 complete [10493.647096] LustreError: 358496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394020 with bad export cookie 4348442367502433550 [10493.651266] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10493.659145] LustreError: 358496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10494.139354] Lustre: server umount lustre-MDT0001 complete [10508.605836] Lustre: server umount lustre-OST0000 complete [10521.196185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.550693] LustreError: 362226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10521.643129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10525.509581] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10526.696092] LustreError: 362227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10531.816423] LustreError: 362226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.724041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10534.086828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10538.332621] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10544.006709] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10548.836305] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10555.304760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10555.586217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10560.839306] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10566.895670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10567.696208] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10580.453150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10580.468643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10580.485246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10580.490776] Lustre: Skipped 2 previous similar messages [10582.500202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10582.509334] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.746784] Lustre: server umount lustre-MDT0000 complete [10587.622364] LustreError: 364064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.642543] LustreError: 364064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10588.712831] LustreError: 362208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394115 with bad export cookie 4348442367502434663 [10588.715970] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10588.728416] LustreError: 362208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10588.936610] Lustre: server umount lustre-MDT0001 complete [10602.968619] Lustre: server umount lustre-OST0000 complete [10618.390598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.899692] LustreError: 366134:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10619.002690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10623.809101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10632.541311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.909377] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10645.937940] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10651.535810] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10658.653256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.959161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.965615] Lustre: Skipped 1 previous similar message [10664.055650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10665.444822] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10673.131342] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10679.777447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10679.785531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.799116] Lustre: Skipped 1 previous similar message [10679.804744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10679.809573] Lustre: Skipped 2 previous similar messages [10682.857741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10684.153819] Lustre: server umount lustre-OST0000 complete [10687.972950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10687.981196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10687.984940] Lustre: Skipped 2 previous similar messages [10687.989192] Lustre: Skipped 2 previous similar messages [10702.304819] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10702.543622] Lustre: server umount lustre-MDT0000 complete [10703.332268] LustreError: 366135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.345465] LustreError: 366135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10706.458684] LustreError: 366115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394233 with bad export cookie 4348442367502435685 [10706.460858] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10706.476481] LustreError: 366115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10706.951344] Lustre: server umount lustre-MDT0001 complete [10718.957743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10722.041335] Key type lgssc unregistered [10722.383270] LNet: 369669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10722.392348] LNetError: 369669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10723.433197] LNet: Removed LNI 192.168.204.144@tcp [10724.227389] Key type .llcrypt unregistered [10724.231337] Key type ._llcrypt unregistered [10750.986079] Key type ._llcrypt registered [10750.994737] Key type .llcrypt registered [10751.166661] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10752.157158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10752.286072] alg: No test for adler32 (adler32-zlib) [10753.326712] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [10753.509560] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10755.200424] Key type lgssc registered [10756.460930] Lustre: Echo OBD driver; http://www.lustre.org/ [10764.473407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10764.498180] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.505677] LustreError: 371110:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10764.519201] LustreError: 371110:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10764.532229] LustreError: 371110:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10764.547441] LustreError: 371110:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10764.557358] LustreError: 371110:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10772.102766] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10772.110536] LustreError: 371508:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10772.126682] LustreError: 371508:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10772.135651] LustreError: 371508:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10772.144277] LustreError: 371508:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10772.153051] LustreError: 371508:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10790.786246] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [10793.822263] Key type lgssc unregistered [10794.213845] LNet: 372512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10794.222670] LNetError: 372512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10794.245945] LNet: Removed LNI 192.168.204.144@tcp [10795.053760] Key type .llcrypt unregistered [10795.055820] Key type ._llcrypt unregistered [10814.941490] Key type ._llcrypt registered [10814.943326] Key type .llcrypt registered [10815.033950] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [10829.067348] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [10830.183663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10830.208471] alg: No test for adler32 (adler32-zlib) [10831.227967] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [10831.498542] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [10833.225260] Key type lgssc registered [10834.135884] Lustre: Echo OBD driver; http://www.lustre.org/ [10867.049522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10867.104630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.457240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10868.481715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10868.561818] Lustre: lustre-MDT0000: new disk, initializing [10868.723884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10868.734121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10872.992678] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10882.662040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10882.789634] Lustre: 375670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10882.856532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10882.863224] Lustre: Skipped 1 previous similar message [10882.969279] Lustre: lustre-MDT0001: new disk, initializing [10883.140545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10883.230775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10883.243208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10888.984634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10901.551148] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10907.911313] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10916.716360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.150897] Lustre: lustre-OST0000: new disk, initializing [10917.154720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10917.161868] Lustre: 377637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10917.291522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10924.333135] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10926.131308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10926.141996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10926.252714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10933.968406] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10939.801620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10939.997934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10945.288463] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10945.481923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10952.167985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10952.175055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10952.194577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10952.207113] Lustre: Skipped 1 previous similar message [10953.965771] Lustre: server umount lustre-OST0000 complete [10957.283203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.293339] Lustre: Skipped 1 previous similar message [10957.296372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10957.299285] Lustre: Skipped 1 previous similar message [10962.407502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10962.413416] Lustre: Skipped 1 previous similar message [10963.600484] Lustre: server umount lustre-MDT0000 complete [10967.465285] LustreError: 375662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394494 with bad export cookie 14529926746540775072 [10967.483160] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10967.930074] Lustre: server umount lustre-MDT0001 complete [10976.927971] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10978.787658] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 06:28:24 (1787394504) [10985.525371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10986.104595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10990.266254] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [10991.586311] LustreError: 380098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10991.600056] LustreError: 380098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10996.707173] LustreError: 380097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.969889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.370493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11004.219628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11011.255431] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11017.661562] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11025.158078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11025.571216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11031.527111] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11038.362457] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11044.826282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11044.996584] Lustre: lustre-OST0001: new disk, initializing [11045.000425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11045.009230] Lustre: 382938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11045.098602] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11051.216239] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11053.121374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11053.135172] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11053.216883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11060.385171] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11063.780102] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11063.792625] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11063.798844] Lustre: Skipped 1 previous similar message [11063.806462] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11063.810598] LustreError: Skipped 1 previous similar message [11063.824234] Lustre: Skipped 1 previous similar message [11068.908166] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11068.917412] Lustre: Skipped 2 previous similar messages [11069.881787] Lustre: server umount lustre-OST0001 complete [11072.308913] LustreError: 381923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.145906] LustreError: 381921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.176248] LustreError: 381921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11084.271908] LustreError: 381923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.284819] LustreError: 381923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11088.355666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11088.361602] LustreError: Skipped 1 previous similar message [11088.364259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11088.371558] Lustre: Skipped 1 previous similar message [11088.379079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11094.398894] Lustre: server umount lustre-OST0000 complete [11099.617757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11099.635514] Lustre: Skipped 2 previous similar messages [11104.001671] Lustre: server umount lustre-MDT0000 complete [11104.739650] LustreError: 382233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.754339] LustreError: 382233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11107.307584] LustreError: 380079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394634 with bad export cookie 14529926746540775954 [11107.320714] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11107.327475] LustreError: 380079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11113.667929] Lustre: server umount lustre-MDT0001 complete [11123.546684] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11126.189507] Key type lgssc unregistered [11126.468728] LNet: 384920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11126.479967] LNetError: 384920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11126.500261] LNet: Removed LNI 192.168.204.144@tcp [11127.464201] Key type .llcrypt unregistered [11127.468672] Key type ._llcrypt unregistered [11174.244447] Key type ._llcrypt registered [11174.246839] Key type .llcrypt registered [11174.379629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11175.592694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11175.760851] alg: No test for adler32 (adler32-zlib) [11176.858807] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11177.136110] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11178.848459] Key type lgssc registered [11180.027469] Lustre: Echo OBD driver; http://www.lustre.org/ [11187.157427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11187.184795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.395981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11188.411838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11188.581876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11192.965517] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11199.815335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.955833] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11199.996227] Lustre: 387731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11200.368800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11204.576522] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11210.950673] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11215.960460] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11220.962399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11220.967145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.000861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11221.473967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.463100] Lustre: server umount lustre-MDT0000 complete [11228.106746] LustreError: 387722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394755 with bad export cookie 14198940942446431016 [11228.109943] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.117156] LustreError: 387722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11228.503550] Lustre: server umount lustre-MDT0001 complete [11237.312734] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 06:32:43 (1787394763) [11243.994331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11244.563908] LustreError: 390239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11244.584728] LustreError: 390239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11244.638895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11249.634328] LustreError: 390240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11249.734292] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11254.758300] LustreError: 390239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.619082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11259.144478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11264.400819] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11270.560913] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11276.036805] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11282.338496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11282.576508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11282.827602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11289.596816] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11296.737350] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11303.228823] Lustre: Permanently deactivating lustre-OST0000 [11303.257898] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11310.654956] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11312.207193] LustreError: 393224:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11312.216449] LustreError: 393224:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11312.225852] LustreError: 393224:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11314.885743] LustreError: 393272:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11314.893445] LustreError: 393272:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11322.862928] Lustre: server umount lustre-OST0000 complete [11325.506126] LustreError: 393523:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11325.513521] LustreError: 393523:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11331.041608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.049374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.049773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.049777] Lustre: Skipped 2 previous similar messages [11331.055860] Lustre: Skipped 1 previous similar message [11336.165208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.171275] Lustre: Skipped 2 previous similar messages [11337.258724] Lustre: server umount lustre-MDT0000 complete [11341.069850] LustreError: 392929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394868 with bad export cookie 14198940942446431590 [11341.072320] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11341.077696] LustreError: 392929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11351.702514] Lustre: server umount lustre-MDT0001 complete [11359.789487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.424023] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11365.633292] LustreError: 394680:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11370.593477] Lustre: 395019:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [11372.553427] Lustre: server umount MGS complete [11383.801340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11384.451692] LustreError: 395744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.492557] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11384.497930] Lustre: Skipped 1 previous similar message [11384.550449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11389.330242] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11389.926398] LustreError: 395745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11397.740600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11398.211729] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11402.499382] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11409.692954] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11415.036650] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11417.232923] Lustre: Permanently reactivating lustre-OST0000 [11417.275153] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11417.286462] Lustre: Skipped 2 previous similar messages [11445.728693] LustreError: 397524:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11452.356684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11452.665467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11452.674847] Lustre: Skipped 1 previous similar message [11459.150952] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11466.403679] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11471.331230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11471.341607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.367829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11471.374558] Lustre: Skipped 1 previous similar message [11473.315925] Lustre: server umount lustre-OST0000 complete [11480.033715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11480.041721] LustreError: Skipped 1 previous similar message [11480.044616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11480.054889] Lustre: Skipped 1 previous similar message [11480.063189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11483.704383] Lustre: server umount lustre-MDT0000 complete [11486.692644] LustreError: 395744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11486.714352] LustreError: 395744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11487.275159] LustreError: 395724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395014 with bad export cookie 14198940942446432647 [11487.281877] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11487.295856] LustreError: 395724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11487.628789] Lustre: server umount lustre-MDT0001 complete [11497.335026] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11500.172992] Key type lgssc unregistered [11500.630671] LNet: 400157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11500.649984] LNetError: 400157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11500.723691] LNet: Removed LNI 192.168.204.144@tcp [11501.710708] Key type .llcrypt unregistered [11501.713959] Key type ._llcrypt unregistered [11518.014401] Key type ._llcrypt registered [11518.017276] Key type .llcrypt registered [11518.120972] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [11534.600729] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11535.723134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11535.910230] alg: No test for adler32 (adler32-zlib) [11536.966522] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11537.193329] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11538.936284] Key type lgssc registered [11540.111420] Lustre: Echo OBD driver; http://www.lustre.org/ [11574.163485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11574.202873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.463550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11575.504663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11575.578774] Lustre: lustre-MDT0000: new disk, initializing [11575.650892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11575.678896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11580.357528] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11590.795084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.901864] Lustre: 403316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11590.964097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11590.968334] Lustre: Skipped 1 previous similar message [11591.052372] Lustre: lustre-MDT0001: new disk, initializing [11591.136479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11591.158494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11591.168153] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11595.921807] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11604.980690] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11609.927115] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11616.370527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11616.787602] Lustre: lustre-OST0000: new disk, initializing [11616.792985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11616.807967] Lustre: 405288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11616.940380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11623.907373] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11625.112805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11625.128106] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11625.311403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11634.006617] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11639.229552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11639.512269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11644.208449] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11644.491295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11651.043620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11651.045568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.055478] LustreError: Skipped 1 previous similar message [11651.056870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11651.095316] Lustre: Skipped 1 previous similar message [11652.831472] Lustre: server umount lustre-OST0000 complete [11656.167871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.187197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11656.195728] Lustre: Skipped 2 previous similar messages [11661.281830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11661.287060] Lustre: Skipped 1 previous similar message [11662.361905] Lustre: server umount lustre-MDT0000 complete [11665.629115] LustreError: 404112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395192 with bad export cookie 8542492520051061415 [11665.630580] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11665.638968] LustreError: 404112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11665.954695] Lustre: server umount lustre-MDT0001 complete [11673.613790] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11675.126724] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:40:00 (1787395200) [11681.284630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11681.844741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11686.451808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11686.881664] LustreError: 407748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.913529] LustreError: 407748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11690.981162] LustreError: 407747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.694299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.048225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11700.066628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11705.896050] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11710.875885] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11717.331538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11717.733707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11724.123536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11732.315689] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11734.310747] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11737.197300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11742.693348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11742.701277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.722558] Lustre: Skipped 1 previous similar message [11742.727803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11746.788980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.799845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11746.837686] Lustre: Skipped 1 previous similar message [11748.873531] Lustre: server umount lustre-OST0000 complete [11757.026660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.035878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11757.042085] Lustre: Skipped 1 previous similar message [11757.059089] Lustre: Skipped 1 previous similar message [11767.264699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11767.657393] Lustre: server umount lustre-MDT0000 complete [11771.361843] LustreError: 409789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.390528] LustreError: 409789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11771.968238] LustreError: 407728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395299 with bad export cookie 8542492520051062297 [11771.976714] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11771.990923] LustreError: 407728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11772.650817] Lustre: server umount lustre-MDT0001 complete [11782.739762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11785.371695] Key type lgssc unregistered [11785.624116] LNet: 411472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11785.635332] LNetError: 411472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11785.669365] LNet: Removed LNI 192.168.204.144@tcp [11786.594495] Key type .llcrypt unregistered [11786.599033] Key type ._llcrypt unregistered [11805.693624] Key type ._llcrypt registered [11805.696661] Key type .llcrypt registered [11805.824482] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11806.076443] Key type .llcrypt unregistered [11806.079288] Key type ._llcrypt unregistered [11822.337486] Key type ._llcrypt registered [11822.341580] Key type .llcrypt registered [11822.443575] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11823.494512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11823.526713] alg: No test for adler32 (adler32-zlib) [11824.551272] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11824.757778] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11826.408173] Key type lgssc registered [11827.671833] Lustre: Echo OBD driver; http://www.lustre.org/ [11835.331758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11835.353993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.888902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11842.279614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11846.128707] LustreError: 413685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.144330] LustreError: 413685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11849.777036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11850.303054] LustreError: 413686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.326190] LustreError: 413686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11850.339227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11855.354795] LustreError: 413685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.399869] LustreError: 413685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11857.090469] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11860.456329] LustreError: 413686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.468736] LustreError: 413686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11861.517439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11865.264174] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11866.595307] LustreError: 413681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.631890] LustreError: 413681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11872.849869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.349528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11877.876843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [11887.074928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11887.083232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.095016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11888.617744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11888.626352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11888.631225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11892.696180] Lustre: server umount lustre-OST0000 complete [11898.850494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11898.853410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.874523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11898.881335] Lustre: Skipped 2 previous similar messages [11902.504274] Lustre: server umount lustre-MDT0000 complete [11903.969267] LustreError: 415422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.986139] LustreError: 415422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11906.292631] LustreError: 413666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395433 with bad export cookie 9074141700144606557 [11906.293947] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11906.308211] LustreError: 413666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11906.770576] Lustre: server umount lustre-MDT0001 complete [11919.115944] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [11922.384732] Key type lgssc unregistered [11922.889481] LNet: 416834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11922.904280] LNetError: 416834:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11922.942496] LNet: Removed LNI 192.168.204.144@tcp [11924.040241] Key type .llcrypt unregistered [11924.049024] Key type ._llcrypt unregistered [11950.184050] Key type ._llcrypt registered [11950.185879] Key type .llcrypt registered [11950.360664] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [11951.936640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11951.997633] alg: No test for adler32 (adler32-zlib) [11953.083699] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [11953.305916] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [11954.968161] Key type lgssc registered [11956.355489] Lustre: Echo OBD driver; http://www.lustre.org/ [11964.128608] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11964.183330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11979.936737] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11985.056340] LustreError: 418225:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [12011.808594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12018.502246] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12025.616235] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12031.608319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12032.095858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12032.168716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12036.522368] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12037.096365] LustreError: 418249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.122109] LustreError: 418249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12042.215910] LustreError: 418250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12044.144034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.475770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12048.402667] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12055.471115] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12061.049435] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12070.014688] Lustre: server umount lustre-OST0000 complete [12075.492583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12075.501496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.532499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.537961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.543867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.575526] Lustre: Skipped 1 previous similar message [12082.661233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12082.671177] Lustre: Skipped 1 previous similar message [12083.889087] Lustre: server umount lustre-MDT0000 complete [12086.496344] Lustre: 417564:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395597/real 1787395597] req@00000000d4da4597 x1874220002866688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787395613 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12086.566507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12086.608600] LustreError: 419259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.670208] LustreError: 419259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12088.504688] LustreError: 418233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395615 with bad export cookie 1907017752312631252 [12088.520480] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12088.536691] LustreError: 418233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12089.098976] Lustre: server umount lustre-MDT0001 complete [12103.703448] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12106.893379] Key type lgssc unregistered [12107.212835] LNet: 421794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12107.219964] LNetError: 421794:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12107.239533] LNet: Removed LNI 192.168.204.144@tcp [12108.035335] Key type .llcrypt unregistered [12108.037571] Key type ._llcrypt unregistered [12131.606292] Key type ._llcrypt registered [12131.608390] Key type .llcrypt registered [12131.748267] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12132.756785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12132.866147] alg: No test for adler32 (adler32-zlib) [12133.943425] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [12134.283403] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12136.024176] Key type lgssc registered [12137.142285] Lustre: Echo OBD driver; http://www.lustre.org/ [12145.395914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12145.407317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.034739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12151.372453] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12157.422547] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.445057] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12159.627346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.044838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12164.061556] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12170.391809] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12175.837502] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12182.476056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.807902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12188.378194] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12192.059649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12194.841238] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12200.930680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12200.944503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.963645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.467652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.470136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.511403] Lustre: Skipped 2 previous similar messages [12206.016843] Lustre: server umount lustre-MDT0000 complete [12207.588791] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12207.614582] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12208.498864] LustreError: 423192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1787395735 with bad export cookie 1422938012622461558 [12208.511636] LustreError: 423192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12210.379462] LustreError: 423190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395737 with bad export cookie 1422938012622460655 [12210.402098] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12210.410334] LustreError: 423190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12211.685032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12211.705767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.723572] Lustre: Skipped 1 previous similar message [12211.733705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12211.740420] Lustre: Skipped 1 previous similar message [12212.707823] LustreError: 423211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12212.729179] LustreError: 423211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12215.714052] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12215.714782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12215.734768] LustreError: 423210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12215.757777] Lustre: Skipped 1 previous similar message [12216.800573] Lustre: server umount lustre-OST0000 complete [12220.838369] LustreError: 423206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.872611] LustreError: 423206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12224.264971] Lustre: server umount lustre-MDT0001 complete [12234.794516] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12237.347158] Key type lgssc unregistered [12237.597619] LNet: 426897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12237.606832] LNetError: 426897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12237.623562] LNet: Removed LNI 192.168.204.144@tcp [12238.610406] Key type .llcrypt unregistered [12238.615220] Key type ._llcrypt unregistered [12267.366909] Key type ._llcrypt registered [12267.369944] Key type .llcrypt registered [12267.453601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12268.427951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12268.459475] alg: No test for adler32 (adler32-zlib) [12269.511697] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [12269.861855] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12271.576231] Key type lgssc registered [12273.198975] Lustre: Echo OBD driver; http://www.lustre.org/ [12280.848368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12280.869810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.497738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12286.737170] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12292.070679] LustreError: 428315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12292.098257] LustreError: 428315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12295.370598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12295.902733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12300.765407] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12308.959375] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12315.050865] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12322.877268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12323.295814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12327.422365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12328.853217] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12335.495365] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12338.971226] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12338.986459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12338.989933] Lustre: Skipped 1 previous similar message [12342.053621] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [12344.290528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12345.135918] Lustre: server umount lustre-MDT0001 complete [12349.411577] LustreError: 428309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.441316] LustreError: 428309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12352.993697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12353.002219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12353.031742] Lustre: Skipped 1 previous similar message [12353.046742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12353.055302] Lustre: Skipped 1 previous similar message [12354.531095] LustreError: 429319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12357.161741] Lustre: server umount lustre-OST0000 complete [12359.653038] LustreError: 428314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.794791] Lustre: server umount lustre-MDT0000 complete [12377.412637] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12380.460853] Key type lgssc unregistered [12380.863586] LNet: 431997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12380.886491] LNetError: 431997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12380.916694] LNet: Removed LNI 192.168.204.144@tcp [12381.800546] Key type .llcrypt unregistered [12381.805905] Key type ._llcrypt unregistered [12408.030533] Key type ._llcrypt registered [12408.033466] Key type .llcrypt registered [12408.169115] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [12424.029866] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12425.958268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12426.014143] alg: No test for adler32 (adler32-zlib) [12427.326859] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [12427.631700] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12429.464812] Key type lgssc registered [12430.965865] Lustre: Echo OBD driver; http://www.lustre.org/ [12472.022799] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12491.087245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12491.128950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.567380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12492.633315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12492.754661] Lustre: lustre-MDT0000: new disk, initializing [12493.022227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12493.094869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12497.991355] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12515.913161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12516.215311] Lustre: lustre-OST0000: new disk, initializing [12516.219064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12516.227068] Lustre: Skipped 1 previous similar message [12516.237885] Lustre: 437033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12516.315209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12522.899080] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12524.688658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12524.704525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12524.749220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12534.149779] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12550.000728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12550.233285] Lustre: 436015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12550.321255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12550.430472] Lustre: lustre-MDT0001: new disk, initializing [12550.586673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12550.634922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12550.657059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12555.663637] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12565.870274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12566.105653] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12571.775054] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12572.131994] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12580.841333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12580.856349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12580.859785] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12582.227382] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [12582.241273] Lustre: Skipped 1 previous similar message [12585.962577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12585.977835] Lustre: Skipped 1 previous similar message [12591.081508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12591.101682] Lustre: Skipped 2 previous similar messages [12593.120120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12593.419696] Lustre: server umount lustre-MDT0001 complete [12601.323049] LustreError: 438348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12601.344534] LustreError: 438348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12606.447207] LustreError: 436022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.475098] LustreError: 436022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12611.559327] LustreError: 436027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.594849] LustreError: 436027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12612.065546] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12612.078955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12612.097519] Lustre: Skipped 1 previous similar message [12612.110365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12612.116312] Lustre: Skipped 1 previous similar message [12616.674473] LustreError: 438795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.311290] Lustre: server umount lustre-OST0000 complete [12621.793887] LustreError: 436027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12628.990734] Lustre: server umount lustre-MDT0000 complete [12642.898744] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12645.946632] Key type lgssc unregistered [12646.262091] LNet: 440783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12646.274695] LNetError: 440783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12646.322594] LNet: Removed LNI 192.168.204.144@tcp [12647.561925] Key type .llcrypt unregistered [12647.564807] Key type ._llcrypt unregistered [12689.497621] Key type ._llcrypt registered [12689.502256] Key type .llcrypt registered [12689.636655] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [12690.031925] Key type .llcrypt unregistered [12690.034592] Key type ._llcrypt unregistered [12707.147976] Key type ._llcrypt registered [12707.150696] Key type .llcrypt registered [12707.258501] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [12708.585179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12708.666913] alg: No test for adler32 (adler32-zlib) [12709.835602] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [12710.170378] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [12711.857173] Key type lgssc registered [12713.145731] Lustre: Echo OBD driver; http://www.lustre.org/ [12742.611695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12742.677811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12743.896814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12743.923703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12744.010737] Lustre: lustre-MDT0000: new disk, initializing [12744.110059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12744.134449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12748.702574] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12759.806248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.977454] Lustre: 444162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12760.026764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12760.033106] Lustre: Skipped 1 previous similar message [12760.131241] Lustre: lustre-MDT0001: new disk, initializing [12760.251869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12760.296080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12760.314019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12765.416798] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12775.147578] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12780.601359] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12788.175496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.503699] Lustre: lustre-OST0000: new disk, initializing [12788.514743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12788.525915] Lustre: 446128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12788.649521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12795.055940] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12797.487170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12797.499967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12797.607020] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12804.991834] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12813.285063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12813.289085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12813.301689] Lustre: Skipped 1 previous similar message [12813.331223] Lustre: Skipped 2 previous similar messages [12818.408643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12818.418232] Lustre: Skipped 2 previous similar messages [12818.768214] Lustre: server umount lustre-MDT0000 complete [12822.152690] LustreError: 444955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396349 with bad export cookie 17998548022611491590 [12822.170796] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12822.176962] LustreError: 444955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12822.532536] Lustre: server umount lustre-MDT0001 complete [12837.160659] Lustre: server umount lustre-OST0000 complete [12847.314193] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:59:32 (1787396372) [12855.928684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12856.664994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12861.492244] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12861.930079] LustreError: 448159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12861.944193] LustreError: 448159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12866.022294] LustreError: 448160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12870.234301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12870.724941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12875.689220] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12883.923361] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12890.602240] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12912.005039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12912.312417] Lustre: Found index 0 for lustre-OST0000, updating log [12912.562875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12919.752437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [12927.880946] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12927.960459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12933.987554] Lustre: server umount lustre-OST0000 complete [12952.288185] Lustre: 442694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396463/real 1787396463] req@00000000ae69338a x1874220796681472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787396479 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12952.319646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12952.339063] Lustre: Skipped 1 previous similar message [12952.363115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12952.371359] Lustre: Skipped 1 previous similar message [12954.400434] Lustre: 442695:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396465/real 1787396465] req@00000000d86fd13f x1874220796682624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12957.669355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12957.677666] Lustre: Skipped 1 previous similar message [12959.648175] Lustre: 442692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396470/real 1787396470] req@00000000d82ad9f9 x1874220796683776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12960.890284] Lustre: server umount lustre-MDT0000 complete [12962.785822] LustreError: 448159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12962.814461] LustreError: 448159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12964.839273] Lustre: 442692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396475/real 1787396475] req@000000007534c41f x1874220796684928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12965.601742] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396492 with bad export cookie 17998548022611492983 [12965.615043] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12965.617543] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12965.976077] Lustre: server umount lustre-MDT0001 complete [12974.368720] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [12977.269996] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 07:01:42 (1787396502) [13596.359404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13596.820967] LustreError: 454583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13596.842750] LustreError: 454583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13596.901506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13601.114177] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13602.273259] LustreError: 454584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13606.393752] LustreError: 454583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13609.685514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13610.075609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13614.731769] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13621.065378] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13626.140580] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13645.870798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13646.223906] Lustre: Found index 0 for lustre-OST0000, updating log [13646.541156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13651.238939] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [13652.528537] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13659.380813] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13661.712897] Lustre: server umount lustre-OST0000 complete [13666.276844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13666.287057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13666.310697] Lustre: Skipped 2 previous similar messages [13666.325979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13667.362124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13667.368952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13668.400216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13668.410296] Lustre: Skipped 2 previous similar messages [13670.438453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13670.448856] Lustre: Skipped 3 previous similar messages [13671.952785] Lustre: server umount lustre-MDT0000 complete [13672.491224] LustreError: 454580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13672.503404] LustreError: 454580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13675.492176] LustreError: 456443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397202 with bad export cookie 17998548022611494061 [13675.498717] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13675.508549] LustreError: 456443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13675.851927] Lustre: server umount lustre-MDT0001 complete [13703.727654] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [13711.491796] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13716.769281] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [13745.174131] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 07:14:31 (1787397271) [13763.004515] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [13766.245338] Key type lgssc unregistered [13766.495820] LNet: 461203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13766.503630] LNetError: 461203:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13858.678059] LNet: Removed LNI 192.168.204.144@tcp [13859.546488] Key type .llcrypt unregistered [13859.548217] Key type ._llcrypt unregistered [13877.241681] Key type ._llcrypt registered [13877.243463] Key type .llcrypt registered [13877.370982] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [13878.697500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13878.722514] alg: No test for adler32 (adler32-zlib) [13879.904936] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [13880.161856] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [13881.824171] Key type lgssc registered [13883.100621] Lustre: Echo OBD driver; http://www.lustre.org/ [13887.892734] LNet: Added LNI 192.168.204.144@tcp100 [8/256/0/180] [13888.724105] LNet: Added LNI 192.168.204.144@tcp101 [8/256/0/180] [13889.569159] LNet: Added LNI 192.168.204.144@tcp102 [8/256/0/180] [13890.492735] LNet: Added LNI 192.168.204.144@tcp103 [8/256/0/180] [13891.594215] LNet: Added LNI 192.168.204.144@tcp104 [8/256/0/180] [13892.575769] LNet: Added LNI 192.168.204.144@tcp105 [8/256/0/180] [13893.536133] LNet: Added LNI 192.168.204.144@tcp106 [8/256/0/180] [13894.390890] LNet: Added LNI 192.168.204.144@tcp107 [8/256/0/180] [13895.332636] LNet: Added LNI 192.168.204.144@tcp108 [8/256/0/180] [13896.208746] LNet: Added LNI 192.168.204.144@tcp109 [8/256/0/180] [13896.975170] LNet: Added LNI 192.168.204.144@tcp110 [8/256/0/180] [13897.770591] LNet: Added LNI 192.168.204.144@tcp111 [8/256/0/180] [13898.723752] LNet: Added LNI 192.168.204.144@tcp112 [8/256/0/180] [13899.671112] LNet: Added LNI 192.168.204.144@tcp113 [8/256/0/180] [13900.460902] LNet: Added LNI 192.168.204.144@tcp114 [8/256/0/180] [13901.420814] LNet: Added LNI 192.168.204.144@tcp115 [8/256/0/180] [13902.449160] LNet: Added LNI 192.168.204.144@tcp116 [8/256/0/180] [13903.269760] LNet: Added LNI 192.168.204.144@tcp117 [8/256/0/180] [13904.326255] LNet: Added LNI 192.168.204.144@tcp118 [8/256/0/180] [13905.104863] LNet: Added LNI 192.168.204.144@tcp119 [8/256/0/180] [13906.029887] LNet: Added LNI 192.168.204.144@tcp120 [8/256/0/180] [13906.977210] LNet: Added LNI 192.168.204.144@tcp121 [8/256/0/180] [13907.925186] LNet: Added LNI 192.168.204.144@tcp122 [8/256/0/180] [13908.787728] LNet: Added LNI 192.168.204.144@tcp123 [8/256/0/180] [13909.664410] LNet: Added LNI 192.168.204.144@tcp124 [8/256/0/180] [13910.495181] LNet: Added LNI 192.168.204.144@tcp125 [8/256/0/180] [13911.401668] LNet: Added LNI 192.168.204.144@tcp126 [8/256/0/180] [13912.178365] LNet: Added LNI 192.168.204.144@tcp127 [8/256/0/180] [13912.912517] LNet: Added LNI 192.168.204.144@tcp128 [8/256/0/180] [13913.595659] LNet: Added LNI 192.168.204.144@tcp129 [8/256/0/180] [13914.344664] LNet: Added LNI 192.168.204.144@tcp130 [8/256/0/180] [13915.184324] LNet: Added LNI 192.168.204.144@tcp131 [8/256/0/180] [13915.990249] LNet: Added LNI 192.168.204.144@tcp132 [8/256/0/180] [13916.826863] LNet: Added LNI 192.168.204.144@tcp133 [8/256/0/180] [13917.715887] LNet: Added LNI 192.168.204.144@tcp134 [8/256/0/180] [13918.551090] LNet: Added LNI 192.168.204.144@tcp135 [8/256/0/180] [13924.737159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13924.757572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13925.873915] Lustre: 464236:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [13925.935665] LNetError: 461696:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13925.968075] LNetError: Protocol error connecting to 192.168.204.144@tcp119 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13925.992235] LNetError: Skipped 4 previous similar messages [13926.096667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13926.158580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13926.240325] LNet: 5 local NIs in recovery (showing 5): 192.168.204.144@tcp124, 192.168.204.144@tcp125, 192.168.204.144@tcp127, 192.168.204.144@tcp126, 192.168.204.144@tcp119 [13926.253269] Lustre: lustre-MDT0000: new disk, initializing [13926.341508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13926.359439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13928.294882] LNetError: 461694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13928.303093] LNetError: 461694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [13928.309391] LNetError: Protocol error connecting to 192.168.204.144@tcp124 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13928.319350] LNetError: Skipped 11 previous similar messages [13931.408116] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13932.388325] LNetError: 464259:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13932.409577] LNetError: 464259:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13932.419205] LNetError: Protocol error connecting to 192.168.204.144@tcp124 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13940.582812] LNetError: 464271:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13940.600817] LNetError: Protocol error connecting to 192.168.204.144@tcp124 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13941.671631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13941.807714] Lustre: 464255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13941.894758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13941.906314] Lustre: Skipped 1 previous similar message [13941.997457] Lustre: lustre-MDT0001: new disk, initializing [13942.044546] Lustre: 465056:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [13942.052682] Lustre: 465056:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13942.257488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13942.350605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13942.376238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13945.524358] Lustre: 465252:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [13945.533188] Lustre: 465252:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13946.736610] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13955.795480] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13956.974774] LNetError: 461695:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13956.998842] LNetError: Protocol error connecting to 192.168.204.144@tcp121 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13962.736943] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13966.179205] LNetError: 461694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13966.203162] LNetError: 461694:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13966.224380] LNetError: Protocol error connecting to 192.168.204.144@tcp128 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13966.248755] LNetError: Skipped 6 previous similar messages [13971.906652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13972.481239] Lustre: lustre-OST0000: new disk, initializing [13972.503747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13972.518676] Lustre: 466232:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13972.828181] Lustre: 466237:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [13972.849974] Lustre: 466237:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13972.987320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13979.208062] Lustre: 466544:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [13979.224275] Lustre: 466544:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13979.325386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13979.342776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13979.625533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13979.763207] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [13984.612924] LNetError: 461695:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [13984.626635] LNetError: 461695:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [13984.639074] LNetError: Protocol error connecting to 192.168.204.144@tcp129 at host 192.168.204.144:988: is it running a compatible version of Lustre? [13984.652348] LNetError: Skipped 12 previous similar messages [13989.266915] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14000.101903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14000.102364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14000.116697] Lustre: Skipped 1 previous similar message [14000.120708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14000.130016] Lustre: Skipped 1 previous similar message [14002.438630] Lustre: server umount lustre-OST0000 complete [14008.801932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14008.811972] LustreError: Skipped 1 previous similar message [14008.815076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14008.828966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14010.337352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14010.338972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14012.386030] Lustre: server umount lustre-MDT0000 complete [14016.296310] LustreError: 466231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397543 with bad export cookie 9341506397853701994 [14016.305153] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14016.306976] LustreError: 466231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14016.716930] Lustre: server umount lustre-MDT0001 complete [14020.450437] LNetError: 464258:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.144 [14020.466239] LNetError: 464258:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [14020.480268] LNetError: Protocol error connecting to 192.168.204.144@tcp121 at host 192.168.204.144:988: is it running a compatible version of Lustre? [14020.497931] LNetError: Skipped 16 previous similar messages [14027.726884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14031.395259] Key type lgssc unregistered [14031.864715] LNet: 468146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14031.874895] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14032.933702] LNet: Removed LNI 192.168.204.144@tcp [14032.940611] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14032.952187] LNet: Removed LNI 192.168.204.144@tcp100 [14034.016765] LNet: Removed LNI 192.168.204.144@tcp101 [14034.028168] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14034.043103] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14034.052911] LNet: Removed LNI 192.168.204.144@tcp102 [14035.104176] LNet: Removed LNI 192.168.204.144@tcp103 [14036.128223] LNet: Removed LNI 192.168.204.144@tcp104 [14036.138696] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14036.144063] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14036.156807] LNet: Removed LNI 192.168.204.144@tcp105 [14037.216270] LNet: Removed LNI 192.168.204.144@tcp106 [14038.240206] LNet: Removed LNI 192.168.204.144@tcp107 [14039.265560] LNet: Removed LNI 192.168.204.144@tcp108 [14040.288256] LNet: Removed LNI 192.168.204.144@tcp109 [14040.299570] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14040.312156] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14040.320907] LNet: Removed LNI 192.168.204.144@tcp110 [14041.377761] LNet: Removed LNI 192.168.204.144@tcp111 [14042.400198] LNet: Removed LNI 192.168.204.144@tcp112 [14043.426089] LNet: Removed LNI 192.168.204.144@tcp113 [14044.448208] LNet: Removed LNI 192.168.204.144@tcp114 [14045.472191] LNet: Removed LNI 192.168.204.144@tcp115 [14046.496219] LNet: Removed LNI 192.168.204.144@tcp116 [14047.523114] LNet: Removed LNI 192.168.204.144@tcp117 [14048.544175] LNet: Removed LNI 192.168.204.144@tcp118 [14048.551307] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14048.560670] LNetError: 468146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14048.566256] LNet: Removed LNI 192.168.204.144@tcp119 [14049.632314] LNet: Removed LNI 192.168.204.144@tcp120 [14050.657776] LNet: Removed LNI 192.168.204.144@tcp121 [14051.680353] LNet: Removed LNI 192.168.204.144@tcp122 [14052.704213] LNet: Removed LNI 192.168.204.144@tcp123 [14053.728253] LNet: Removed LNI 192.168.204.144@tcp124 [14054.752145] LNet: Removed LNI 192.168.204.144@tcp125 [14055.776942] LNet: Removed LNI 192.168.204.144@tcp126 [14056.800205] LNet: Removed LNI 192.168.204.144@tcp127 [14057.825200] LNet: Removed LNI 192.168.204.144@tcp128 [14058.848584] LNet: Removed LNI 192.168.204.144@tcp129 [14058.855348] LNet: Removed LNI 192.168.204.144@tcp130 [14058.862108] LNet: Removed LNI 192.168.204.144@tcp131 [14058.871633] LNet: Removed LNI 192.168.204.144@tcp132 [14058.877317] LNet: Removed LNI 192.168.204.144@tcp133 [14058.883137] LNet: Removed LNI 192.168.204.144@tcp134 [14058.893830] LNet: Removed LNI 192.168.204.144@tcp135 [14059.859608] Key type .llcrypt unregistered [14059.861768] Key type ._llcrypt unregistered [14086.852968] Key type ._llcrypt registered [14086.856878] Key type .llcrypt registered [14087.015454] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14106.081322] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14107.749310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14107.810996] alg: No test for adler32 (adler32-zlib) [14108.997389] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [14109.364542] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14111.168471] Key type lgssc registered [14112.716132] Lustre: Echo OBD driver; http://www.lustre.org/ [14153.670475] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 07:21:19 (1787397679) [14172.502602] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14175.967370] Key type lgssc unregistered [14176.337799] LNet: 472238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14176.346615] LNetError: 472238:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14176.364176] LNet: Removed LNI 192.168.204.144@tcp [14177.448873] Key type .llcrypt unregistered [14177.455722] Key type ._llcrypt unregistered [14195.107624] Key type ._llcrypt registered [14195.111901] Key type .llcrypt registered [14195.253620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14196.462679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14196.554932] alg: No test for adler32 (adler32-zlib) [14197.674846] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [14197.919341] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14199.634755] Key type lgssc registered [14200.867377] Lustre: Echo OBD driver; http://www.lustre.org/ [14210.925251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14210.957918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14212.299136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14212.368733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14212.516771] Lustre: lustre-MDT0000: new disk, initializing [14212.634187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14212.649720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14218.370635] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14230.536551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14230.708607] Lustre: 473450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14230.807303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14230.813129] Lustre: Skipped 1 previous similar message [14230.916501] Lustre: lustre-MDT0001: new disk, initializing [14231.032303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14231.089759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14231.114696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14236.385539] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14247.050805] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14252.987466] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14260.254507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14260.521250] Lustre: lustre-OST0000: new disk, initializing [14260.523840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14260.528863] Lustre: 475421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14260.609333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14266.960869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14266.976118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14267.163894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14267.510357] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14278.240137] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14283.651628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lnet_if_list [14290.841384] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14290.891748] LNet: Added LNI 192.168.204.144@tcp10 [8/256/0/180] [14298.352350] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14298.399700] LNet: Added LNI 192.168.204.144@tcp11 [8/256/0/180] [14303.465200] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14303.489127] LNet: Added LNI 192.168.204.144@tcp12 [8/256/0/180] [14308.919724] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14308.945541] LNet: Added LNI 192.168.204.144@tcp13 [8/256/0/180] [14314.837367] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14314.866734] LNet: Added LNI 192.168.204.144@tcp14 [8/256/0/180] [14320.343539] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14320.378630] LNet: Added LNI 192.168.204.144@tcp15 [8/256/0/180] [14326.054833] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14326.094378] LNet: Added LNI 192.168.204.144@tcp16 [8/256/0/180] [14332.555930] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14332.617065] LNet: Added LNI 192.168.204.144@tcp17 [8/256/0/180] [14338.965564] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14339.000053] LNet: Added LNI 192.168.204.144@tcp18 [8/256/0/180] [14345.426734] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14345.458411] LNet: Added LNI 192.168.204.144@tcp19 [8/256/0/180] [14351.861637] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14351.906302] LNet: Added LNI 192.168.204.144@tcp20 [8/256/0/180] [14358.185341] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14358.234823] LNet: Added LNI 192.168.204.144@tcp21 [8/256/0/180] [14364.686669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14364.725816] LNet: Added LNI 192.168.204.144@tcp22 [8/256/0/180] [14371.539924] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14371.568819] LNet: Added LNI 192.168.204.144@tcp23 [8/256/0/180] [14377.628432] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14377.651454] LNet: Added LNI 192.168.204.144@tcp24 [8/256/0/180] [14384.929118] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14384.956380] LNet: Added LNI 192.168.204.144@tcp25 [8/256/0/180] [14391.451995] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14391.525325] LNet: Added LNI 192.168.204.144@tcp26 [8/256/0/180] [14398.909276] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14398.960393] LNet: Added LNI 192.168.204.144@tcp27 [8/256/0/180] [14405.395403] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14405.435575] LNet: Added LNI 192.168.204.144@tcp28 [8/256/0/180] [14411.715316] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14411.756436] LNet: Added LNI 192.168.204.144@tcp29 [8/256/0/180] [14418.103921] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14418.156404] LNet: Added LNI 192.168.204.144@tcp30 [8/256/0/180] [14425.741703] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14425.789644] LNet: Added LNI 192.168.204.144@tcp31 [8/256/0/180] [14433.108058] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14433.160713] LNet: Added LNI 192.168.204.144@tcp32 [8/256/0/180] [14441.200830] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14441.229173] LNet: Added LNI 192.168.204.144@tcp33 [8/256/0/180] [14448.614394] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14448.653415] LNet: Added LNI 192.168.204.144@tcp34 [8/256/0/180] [14458.060210] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14458.088300] LNet: Added LNI 192.168.204.144@tcp35 [8/256/0/180] [14467.700921] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14467.763609] LNet: Added LNI 192.168.204.144@tcp36 [8/256/0/180] [14475.973169] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14476.012351] LNet: Added LNI 192.168.204.144@tcp37 [8/256/0/180] [14484.505424] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14484.544809] LNet: Added LNI 192.168.204.144@tcp38 [8/256/0/180] [14493.370827] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14493.397958] LNet: Added LNI 192.168.204.144@tcp39 [8/256/0/180] [14501.659886] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14501.702414] LNet: Added LNI 192.168.204.144@tcp40 [8/256/0/180] [14510.695520] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14510.728089] LNet: Added LNI 192.168.204.144@tcp41 [8/256/0/180] [14519.956321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14519.989859] LNet: Added LNI 192.168.204.144@tcp42 [8/256/0/180] [14529.487525] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14529.529497] LNet: Added LNI 192.168.204.144@tcp43 [8/256/0/180] [14539.147433] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14539.216885] LNet: Added LNI 192.168.204.144@tcp44 [8/256/0/180] [14550.166965] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14550.219931] LNet: Added LNI 192.168.204.144@tcp45 [8/256/0/180] [14559.083759] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14559.129493] LNet: Added LNI 192.168.204.144@tcp46 [8/256/0/180] [14568.559299] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14568.591574] LNet: Added LNI 192.168.204.144@tcp47 [8/256/0/180] [14580.032440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14580.074744] LNet: Added LNI 192.168.204.144@tcp48 [8/256/0/180] [14589.762144] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14589.803839] LNet: Added LNI 192.168.204.144@tcp49 [8/256/0/180] [14600.944316] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14601.013484] LNet: Added LNI 192.168.204.144@tcp50 [8/256/0/180] [14625.254499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14625.272917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14625.318888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14629.593070] Lustre: server umount lustre-MDT0000 complete [14633.878078] LustreError: 473441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398160 with bad export cookie 8754372355275432326 [14633.883483] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14633.886310] LustreError: 473441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14634.194207] Lustre: server umount lustre-MDT0001 complete [14652.197109] Lustre: 472724:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398163/real 1787398163] req@0000000077fabf42 x1874222356029312/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787398179 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14652.215548] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14652.227323] Lustre: Skipped 2 previous similar messages [14653.529620] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14654.434281] Lustre: 472723:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398165/real 1787398165] req@0000000014f1638c x1874222356029440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787398181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14657.570124] Lustre: 472723:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398168/real 1787398168] req@000000008a7b492c x1874222356029568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787398184 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14663.654847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14664.672197] Lustre: 472725:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398175/real 1787398175] req@000000003dfb41e8 x1874222356029952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787398191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14664.728264] Lustre: 472725:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14678.885127] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x797dc74643b12bdc [14678.896549] Lustre: MGC192.168.204.144@tcp: Connection restored to 0@lo (at 0@lo) [14679.221066] LustreError: 475414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14679.245946] LustreError: 475414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14679.287103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14681.318702] LustreError: 475415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14681.340950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14682.345023] LustreError: 475414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14685.426366] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14687.716740] LustreError: 475415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14692.841159] LustreError: 475414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14696.001597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14696.391171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14697.462072] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14702.205967] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14711.210980] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14718.194701] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14740.970351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14740.976068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14740.995458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14741.006565] Lustre: Skipped 2 previous similar messages [14742.505758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14742.520962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14748.647540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14748.652388] Lustre: Skipped 3 previous similar messages [14753.248174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14753.395780] Lustre: server umount lustre-OST0000 complete [14758.891847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14758.898226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14758.928236] Lustre: Skipped 2 previous similar messages [14758.941327] Lustre: Skipped 1 previous similar message [14764.340604] Lustre: server umount lustre-MDT0000 complete [14768.586567] LustreError: 475420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398295 with bad export cookie 8754372355275434972 [14768.599553] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14768.602282] LustreError: 475420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14769.389449] Lustre: server umount lustre-MDT0001 complete [14783.993208] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [14788.380442] Key type lgssc unregistered [14789.007282] LNet: 502475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14789.021728] LNetError: 502475:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14789.030126] LNet: Removed LNI 192.168.204.144@tcp [14789.031951] LNet: Removed LNI 192.168.204.144@tcp10 [14789.047182] LNet: Removed LNI 192.168.204.144@tcp11 [14789.055468] LNet: Removed LNI 192.168.204.144@tcp12 [14789.067920] LNet: Removed LNI 192.168.204.144@tcp13 [14789.073940] LNet: Removed LNI 192.168.204.144@tcp14 [14789.077285] LNet: Removed LNI 192.168.204.144@tcp15 [14789.079287] LNet: Removed LNI 192.168.204.144@tcp16 [14789.082287] LNet: Removed LNI 192.168.204.144@tcp17 [14789.090985] LNet: Removed LNI 192.168.204.144@tcp18 [14789.099097] LNet: Removed LNI 192.168.204.144@tcp19 [14789.103495] LNet: Removed LNI 192.168.204.144@tcp20 [14789.107563] LNet: Removed LNI 192.168.204.144@tcp21 [14789.120408] LNet: Removed LNI 192.168.204.144@tcp22 [14789.122623] LNet: Removed LNI 192.168.204.144@tcp23 [14789.137954] LNet: Removed LNI 192.168.204.144@tcp24 [14789.142934] LNet: Removed LNI 192.168.204.144@tcp25 [14789.146558] LNet: Removed LNI 192.168.204.144@tcp26 [14789.154797] LNet: Removed LNI 192.168.204.144@tcp27 [14789.161642] LNet: Removed LNI 192.168.204.144@tcp28 [14789.171535] LNet: Removed LNI 192.168.204.144@tcp29 [14789.177388] LNet: Removed LNI 192.168.204.144@tcp30 [14789.186352] LNet: Removed LNI 192.168.204.144@tcp31 [14789.194394] LNet: Removed LNI 192.168.204.144@tcp32 [14789.202929] LNet: Removed LNI 192.168.204.144@tcp33 [14789.204872] LNet: Removed LNI 192.168.204.144@tcp34 [14789.221565] LNet: Removed LNI 192.168.204.144@tcp35 [14789.223649] LNet: Removed LNI 192.168.204.144@tcp36 [14789.231919] LNet: Removed LNI 192.168.204.144@tcp37 [14789.244913] LNet: Removed LNI 192.168.204.144@tcp38 [14789.247021] LNet: Removed LNI 192.168.204.144@tcp39 [14789.257198] LNet: Removed LNI 192.168.204.144@tcp40 [14789.267713] LNet: Removed LNI 192.168.204.144@tcp41 [14789.269668] LNet: Removed LNI 192.168.204.144@tcp42 [14789.278611] LNet: Removed LNI 192.168.204.144@tcp43 [14789.280552] LNet: Removed LNI 192.168.204.144@tcp44 [14789.292189] LNet: Removed LNI 192.168.204.144@tcp45 [14789.298950] LNet: Removed LNI 192.168.204.144@tcp46 [14789.311286] LNet: Removed LNI 192.168.204.144@tcp47 [14789.315451] LNet: Removed LNI 192.168.204.144@tcp48 [14789.317442] LNet: Removed LNI 192.168.204.144@tcp49 [14789.340808] LNet: Removed LNI 192.168.204.144@tcp50 [14790.232359] Key type .llcrypt unregistered [14790.241921] Key type ._llcrypt unregistered [14814.582344] Key type ._llcrypt registered [14814.588883] Key type .llcrypt registered [14814.696245] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [14834.789547] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [14836.122746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14836.307875] alg: No test for adler32 (adler32-zlib) [14837.674331] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [14837.848520] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [14839.504967] Key type lgssc registered [14841.342947] Lustre: Echo OBD driver; http://www.lustre.org/ [14880.836909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14880.849533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14882.217962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14882.371323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14882.602663] Lustre: lustre-MDT0000: new disk, initializing [14882.912715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14882.959314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14889.171846] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14901.813399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14901.983608] Lustre: 505676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14902.071949] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14902.077730] Lustre: Skipped 1 previous similar message [14902.178558] Lustre: lustre-MDT0001: new disk, initializing [14902.274258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14902.310113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14902.322787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14907.493817] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14918.236288] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14925.012958] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14932.936145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14933.158450] Lustre: *** cfs_fail_loc=90f, val=0*** [14933.260609] Lustre: lustre-OST0000: new disk, initializing [14933.265824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14933.275512] Lustre: 507692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14933.389404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14940.788279] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [14942.354682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14942.361739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14942.499836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14950.670781] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14958.050064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14958.057804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14958.059913] LustreError: Skipped 1 previous similar message [14958.074620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14958.106309] Lustre: Skipped 1 previous similar message [14963.169703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14963.185194] Lustre: Skipped 2 previous similar messages [14968.294968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14968.307444] Lustre: Skipped 1 previous similar message [14972.384129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14972.844539] Lustre: server umount lustre-OST0000 complete [14978.535762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14978.575929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14978.592747] Lustre: Skipped 1 previous similar message [14983.652285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14983.663635] Lustre: Skipped 1 previous similar message [14984.011430] Lustre: server umount lustre-MDT0000 complete [14988.165970] LustreError: 508846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398515 with bad export cookie 5591563811386234248 [14988.170349] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14988.174520] LustreError: 508846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14988.554547] Lustre: server umount lustre-MDT0001 complete [15000.044683] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15002.848739] Key type lgssc unregistered [15003.075619] LNet: 509599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15003.091459] LNetError: 509599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15004.134547] LNet: Removed LNI 192.168.204.144@tcp [15005.379127] Key type .llcrypt unregistered [15005.383212] Key type ._llcrypt unregistered [15028.559303] Key type ._llcrypt registered [15028.561118] Key type .llcrypt registered [15028.793092] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15030.442149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15030.552083] alg: No test for adler32 (adler32-zlib) [15031.679572] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [15031.908701] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [15033.608399] Key type lgssc registered [15035.126080] Lustre: Echo OBD driver; http://www.lustre.org/ [15043.660988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15043.685531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15045.215191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15049.758592] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15055.329966] LustreError: 510870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15055.363114] LustreError: 510870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15059.292203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15059.769258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15065.234569] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15073.582746] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15080.595993] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15088.902815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15089.234127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15097.416435] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15105.608169] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15116.480610] loop0: detected capacity change from 0 to 204800000 [15116.759440] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15116.780225] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15116.799587] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15123.109614] loop0: detected capacity change from 0 to 204800000 [15123.338967] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15123.353131] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15123.368365] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15129.359879] loop0: detected capacity change from 0 to 204800000 [15135.912241] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15136.049745] Lustre: fs15246-OST0000: new disk, initializing [15136.052097] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15136.056705] Lustre: 514463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15136.128358] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15143.767232] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15154.739707] loop1: detected capacity change from 0 to 204800000 [15161.274917] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15161.459816] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15161.507623] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15161.636349] Lustre: fs15246-MDT0000: new disk, initializing [15161.868864] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15161.912177] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15162.043619] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15163.111710] Lustre: 515587:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000097a67367 x1874223230402816/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787398706 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15167.250760] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15172.199279] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15172.216710] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15172.239930] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15190.508049] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15190.536668] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15195.622203] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15196.745660] Lustre: server umount fs15246-MDT0000 complete [15200.758069] LustreError: 510866:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15205.873990] LustreError: 510865:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15207.053181] Lustre: server umount fs15246-OST0000 complete [15221.242925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15221.247468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15221.267875] Lustre: Skipped 1 previous similar message [15221.288390] Lustre: Skipped 1 previous similar message [15225.170983] Lustre: server umount lustre-OST0000 complete [15231.459108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15231.460972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15231.471900] Lustre: Skipped 1 previous similar message [15231.490401] Lustre: Skipped 1 previous similar message [15235.769845] Lustre: server umount lustre-MDT0000 complete [15236.579144] LustreError: 510871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15236.591820] LustreError: 510871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15239.713584] LustreError: 510852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398766 with bad export cookie 13212429972101360274 [15239.729498] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15239.731021] LustreError: 510852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15240.104574] Lustre: server umount lustre-MDT0001 complete [15252.947315] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15255.952335] Key type lgssc unregistered [15256.340705] LNet: 518685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15256.347492] LNetError: 518685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15256.369687] LNet: Removed LNI 192.168.204.144@tcp [15257.357429] Key type .llcrypt unregistered [15257.362805] Key type ._llcrypt unregistered [15278.316631] Key type ._llcrypt registered [15278.319888] Key type .llcrypt registered [15278.474936] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15279.773276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15279.897502] alg: No test for adler32 (adler32-zlib) [15281.102669] Lustre: Lustre: Build Version: 2.17.54_113_ga48d3e3 [15281.308070] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [15282.944243] Key type lgssc registered [15284.021937] Lustre: Echo OBD driver; http://www.lustre.org/ [15325.583915] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [15335.797282] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15373.940682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15373.981663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15375.154595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15375.182503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15375.287664] Lustre: lustre-MDT0000: new disk, initializing [15375.356893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15375.370362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15380.628629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15393.106543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15393.195720] Lustre: 523158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15393.240375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15393.244813] Lustre: Skipped 1 previous similar message [15393.309740] Lustre: lustre-MDT0001: new disk, initializing [15393.389333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15393.423480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15393.438317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15400.116636] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15411.765971] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15419.254354] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15427.849660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15428.210673] Lustre: lustre-OST0000: new disk, initializing [15428.220649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15428.229713] Lustre: 525126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15428.347481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15435.439623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15435.459305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15435.603488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15436.738875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [15447.326974] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15452.998417] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15453.193878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15458.665814] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15458.990792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15461.348369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15461.348723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15461.358496] LustreError: Skipped 1 previous similar message [15461.368372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15466.476166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15466.488789] Lustre: Skipped 2 previous similar messages [15467.676223] Lustre: server umount lustre-OST0000 complete [15475.169703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15475.182317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15475.204488] Lustre: Skipped 1 previous similar message [15475.215582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15476.709561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15478.228974] Lustre: server umount lustre-MDT0000 complete [15482.380648] LustreError: 523952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399009 with bad export cookie 9314396334139464202 [15482.394418] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15482.400342] LustreError: 523952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15482.851800] Lustre: server umount lustre-MDT0001 complete [15511.971461] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [15523.125682] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [15560.283484] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15275 sec ========= 07:44:45 (1787399085) [15562.467046] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:44:47 (1787399087) === [15566.966944] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:44:51 (1787399091) === [15613.268257] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [15616.192828] Key type lgssc unregistered [15616.768814] LNet: 531775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15616.783254] LNetError: 531775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15616.815402] LNet: Removed LNI 192.168.204.144@tcp [15617.810960] Key type .llcrypt unregistered [15617.814972] Key type ._llcrypt unregistered