[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508962058 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003153] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007973] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.011194] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013051] SELinux: Initializing. [ 0.015043] *** VALIDATE selinux *** [ 0.023769] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029321] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030192] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033171] *** VALIDATE tmpfs *** [ 0.035408] *** VALIDATE proc *** [ 0.036503] *** VALIDATE cgroup *** [ 0.038006] *** VALIDATE cgroup2 *** [ 0.039315] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042036] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046680] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.049000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049964] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051036] ... version: 2 [ 0.052024] ... bit width: 48 [ 0.053016] ... generic registers: 4 [ 0.054018] ... value mask: 0000ffffffffffff [ 0.055025] ... max period: 00007fffffffffff [ 0.056022] ... fixed-purpose events: 3 [ 0.057020] ... event mask: 000000070000000f [ 0.058410] rcu: Hierarchical SRCU implementation. [ 0.061929] smp: Bringing up secondary CPUs ... [ 0.063508] x86: Booting SMP configuration: [ 0.064048] .... node #0, CPUs: #1 #2 #3 [ 0.072266] smp: Brought up 1 node, 4 CPUs [ 0.074017] smpboot: Max logical packages: 1 [ 0.075037] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164026] node 0 deferred pages initialised in 85ms [ 0.173128] devtmpfs: initialized [ 0.174834] x86/mm: Memory block size: 128MB [ 0.181000] gcov: version magic: 0x41383552 [ 0.184648] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.189358] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.193937] pinctrl core: initialized pinctrl subsystem [ 0.196469] [ 0.198009] ************************************************************* [ 0.202032] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207026] ** ** [ 0.210014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.213016] ** ** [ 0.217025] ** This means that this kernel is built to expose internal ** [ 0.220016] ** IOMMU data structures, which may compromise security on ** [ 0.224021] ** your system. ** [ 0.227029] ** ** [ 0.230021] ** If you see this message and you are not debugging the ** [ 0.234022] ** kernel, report this immediately to your vendor! ** [ 0.237032] ** ** [ 0.240022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243221] ************************************************************* [ 0.246936] NET: Registered protocol family 16 [ 0.249898] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.254089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.261133] cpuidle: using governor menu [ 0.263624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268027] PCI: Using configuration type 1 for base access [ 0.271201] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.281432] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.282093] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284174] cryptd: max_cpu_qlen set to 1000 [ 0.287313] ACPI: Added _OSI(Module Device) [ 0.288016] ACPI: Added _OSI(Processor Device) [ 0.290017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299342] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.309649] ACPI: Interpreter enabled [ 0.312100] ACPI: PM: (supports S0 S3 S4 S5) [ 0.313021] ACPI: Using IOAPIC for interrupt routing [ 0.316143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320733] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.340210] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.344071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.355121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.364436] acpiphp: Slot [2] registered [ 0.367348] acpiphp: Slot [5] registered [ 0.369775] acpiphp: Slot [6] registered [ 0.372616] acpiphp: Slot [7] registered [ 0.376143] acpiphp: Slot [8] registered [ 0.380370] acpiphp: Slot [9] registered [ 0.383379] acpiphp: Slot [10] registered [ 0.385455] acpiphp: Slot [3] registered [ 0.388309] acpiphp: Slot [4] registered [ 0.390173] acpiphp: Slot [11] registered [ 0.393237] acpiphp: Slot [12] registered [ 0.395144] acpiphp: Slot [13] registered [ 0.396419] acpiphp: Slot [14] registered [ 0.398294] acpiphp: Slot [15] registered [ 0.400166] acpiphp: Slot [16] registered [ 0.402131] acpiphp: Slot [17] registered [ 0.404263] acpiphp: Slot [18] registered [ 0.406172] acpiphp: Slot [19] registered [ 0.408156] acpiphp: Slot [20] registered [ 0.410301] acpiphp: Slot [21] registered [ 0.413154] acpiphp: Slot [22] registered [ 0.415156] acpiphp: Slot [23] registered [ 0.418286] acpiphp: Slot [24] registered [ 0.420181] acpiphp: Slot [25] registered [ 0.422339] acpiphp: Slot [26] registered [ 0.425480] acpiphp: Slot [27] registered [ 0.428379] acpiphp: Slot [28] registered [ 0.430155] acpiphp: Slot [29] registered [ 0.432343] acpiphp: Slot [30] registered [ 0.435230] acpiphp: Slot [31] registered [ 0.437245] PCI host bridge to bus 0000:00 [ 0.438025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.449052] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.455040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.459049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.462190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.466921] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.474568] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.490737] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.496892] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.502596] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.505024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.510034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.515288] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.521774] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.527121] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.531630] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.539023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.555022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.563018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.575432] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.587094] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.600021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.627100] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.646351] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.655022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.669034] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.704025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.725327] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.735021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.745043] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.771072] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.785291] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.797031] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.807029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.832000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.846000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.858020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.870025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.897025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.919809] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.931020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.943023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.973022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.995958] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.002788] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.007807] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.010733] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.014275] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.021173] iommu: Default domain type: Passthrough [ 1.023848] SCSI subsystem initialized [ 1.026303] ACPI: bus type USB registered [ 1.029209] usbcore: registered new interface driver usbfs [ 1.031153] usbcore: registered new interface driver hub [ 1.033200] usbcore: registered new device driver usb [ 1.036414] pps_core: LinuxPPS API ver. 1 registered [ 1.037016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.041089] PTP clock support registered [ 1.045052] EDAC MC: Ver: 3.0.0 [ 1.047152] PCI: Using ACPI for IRQ routing [ 1.049000] NetLabel: Initializing [ 1.050019] NetLabel: domain hash size = 128 [ 1.052020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.055661] NetLabel: unlabeled traffic allowed by default [ 1.060588] vgaarb: loaded [ 1.063213] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.065016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.073476] clocksource: Switched to clocksource kvm-clock [ 1.210044] VFS: Disk quotas dquot_6.6.0 [ 1.211456] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.213984] *** VALIDATE ramfs *** [ 1.215132] *** VALIDATE hugetlbfs *** [ 1.217379] pnp: PnP ACPI init [ 1.219954] pnp: PnP ACPI: found 6 devices [ 1.238216] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.242546] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.245396] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.248207] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.251378] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.254729] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.258090] NET: Registered protocol family 2 [ 1.261258] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.267730] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.272533] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.278458] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.282562] TCP: Hash tables configured (established 65536 bind 65536) [ 1.286107] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.289706] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.292731] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.296444] NET: Registered protocol family 1 [ 1.302357] RPC: Registered named UNIX socket transport module. [ 1.305411] RPC: Registered udp transport module. [ 1.307581] RPC: Registered tcp transport module. [ 1.309951] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.313380] NET: Registered protocol family 44 [ 1.315711] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.318703] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.321583] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.324744] PCI: CLS 0 bytes, default 64 [ 1.326713] Unpacking initramfs... [ 3.510697] debug: unmapping init [mem 0xffff9142bcc54000-0xffff9142bffbffff] [ 3.518692] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.521263] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.527191] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.301662] Initialise system trusted keyrings [ 4.303052] Key type blacklist registered [ 4.305086] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.317840] zbud: loaded [ 4.320710] *** VALIDATE nfs *** [ 4.321726] *** VALIDATE nfs4 *** [ 4.323261] pstore: using deflate compression [ 4.327079] Platform Keyring initialized [ 4.451504] NET: Registered protocol family 38 [ 4.453288] Key type asymmetric registered [ 4.455092] Asymmetric key parser 'x509' registered [ 4.457240] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.460848] io scheduler mq-deadline registered [ 4.462399] io scheduler kyber registered [ 4.464649] io scheduler bfq registered [ 4.466761] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.469976] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.472974] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.476282] ACPI: Power Button [PWRF] [ 4.483491] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.495241] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.512146] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.522615] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.543530] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.576678] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.605080] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.610032] Non-volatile memory driver v1.3 [ 4.611469] Linux agpgart interface v0.103 [ 4.648187] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.651929] vda: detected capacity change from 0 to 76787712 [ 4.667699] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.670665] vdb: detected capacity change from 0 to 1073741824 [ 4.690884] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.695219] vdc: detected capacity change from 0 to 2621440000 [ 4.715193] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.719758] vdd: detected capacity change from 0 to 2621440000 [ 4.739202] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.743528] vde: detected capacity change from 0 to 4294967296 [ 4.762441] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.765120] vdf: detected capacity change from 0 to 4294967296 [ 4.775023] libphy: Fixed MDIO Bus: probed [ 4.780606] usbcore: registered new interface driver usbserial_generic [ 4.783888] usbserial: USB Serial support registered for generic [ 4.787377] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.793432] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.798647] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.800991] mousedev: PS/2 mouse device common for all mice [ 4.804943] rtc_cmos 00:05: RTC can wake from S4 [ 4.808776] rtc_cmos 00:05: registered as rtc0 [ 4.813362] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.815949] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.819624] intel_pstate: CPU model not supported [ 4.828115] hid: raw HID events driver (C) Jiri Kosina [ 4.832625] usbcore: registered new interface driver usbhid [ 4.834648] usbhid: USB HID core driver [ 4.836815] drop_monitor: Initializing network drop monitor service [ 4.839401] Initializing XFRM netlink socket [ 4.843840] NET: Registered protocol family 10 [ 4.844369] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.849034] Segment Routing with IPv6 [ 4.849077] NET: Registered protocol family 17 [ 4.849338] mpls_gso: MPLS GSO support [ 4.863462] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.875666] RAS: Correctable Errors collector initialized. [ 4.881359] AVX version of gcm_enc/dec engaged. [ 4.883603] AES CTR mode by8 optimization enabled [ 5.001213] sched_clock: Marking stable (5001194627, 0)->(6267677958, -1266483331) [ 5.005463] registered taskstats version 1 [ 5.008407] Loading compiled-in X.509 certificates [ 5.011897] zswap: loaded using pool lzo/zbud [ 5.052705] Key type big_key registered [ 5.073396] Key type encrypted registered [ 5.078780] ima: No TPM chip found, activating TPM-bypass! [ 5.080744] ima: Allocated hash algorithm: sha1 [ 5.082503] ima: No architecture policies found [ 5.085391] evm: Initialising EVM extended attributes: [ 5.087900] evm: security.selinux [ 5.089899] evm: security.ima [ 5.090976] evm: security.capability [ 5.092801] evm: HMAC attrs: 0x1 [ 5.096940] rtc_cmos 00:05: setting system clock to 2026-09-17 22:11:12 UTC (1789683072) [ 5.107641] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 5.111749] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 5.124186] Write protecting the kernel read-only data: 28672k [ 5.129720] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 5.132438] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 5.180413] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.191660] systemd[1]: Detected virtualization kvm. [ 5.194070] systemd[1]: Detected architecture x86-64. [ 5.196675] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.229425] systemd[1]: No hostname configured. [ 5.231975] systemd[1]: Set hostname to . [ 5.234814] random: systemd: uninitialized urandom read (16 bytes read) [ 5.238870] systemd[1]: Initializing machine ID from random generator. [ 5.361136] random: ln: uninitialized urandom read (6 bytes read) [ 5.577584] random: systemd: uninitialized urandom read (16 bytes read) [ 5.580484] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.588783] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.592698] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.861672] device-mapper: uevent: version 1.0.3 [ 6.870549] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.933779] virtio_net virtio0 ens2: renamed from eth0 [ 9.072865] random: fast init done [ 9.419700] scsi host0: ata_piix [ 9.522061] scsi host1: ata_piix [ 9.531133] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.541158] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.611424] random: crng init done [ 14.613909] random: 7 urandom warning(s) missed due to ratelimiting [ 16.938913] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.713142] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.885090] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.458362] SELinux: Disabled at runtime. [ 21.543383] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.554527] systemd[1]: Detected virtualization kvm. [ 21.557055] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.437535] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.441858] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.454863] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.461709] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.469893] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.489483] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.498410] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ 22.730014] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 24.207658] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.230175] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.233785] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.995915] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.197522] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.714804] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 33.569336] NFS: Registering the id_resolver key type [ 33.571631] Key type id_resolver registered [ 33.573549] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 47.379408] hrtimer: interrupt took 19883531 ns [ 92.577996] libcfs: loading out-of-tree module taints kernel. [ 92.614187] Key type ._llcrypt registered [ 92.616300] Key type .llcrypt registered [ 92.691295] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 108.889268] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 109.933341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.941938] alg: No test for adler32 (adler32-zlib) [ 111.331673] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 112.177035] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 113.864323] Key type lgssc registered [ 115.274793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.554530] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.217771] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 183.869386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.948729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.580304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.655640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.852943] Lustre: lustre-MDT0000: new disk, initializing [ 186.058587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.093425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 195.583461] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.414338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.577192] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.632940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.642239] Lustre: Skipped 1 previous similar message [ 216.772295] Lustre: lustre-MDT0001: new disk, initializing [ 216.870967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.910188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.933299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.245573] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.688821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 238.520451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.818650] Lustre: lustre-OST0000: new disk, initializing [ 238.822588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.827101] Lustre: 8456:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.893863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.778638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.793641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.877030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.899848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.071122] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.218774] Lustre: lustre-OST0001: new disk, initializing [ 263.231981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.246521] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.360232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.107880] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.930752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 272.948653] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.981579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 282.690855] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 288.862865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 295.820944] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.537883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 308.056589] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 310.766488] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 314.129775] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 316.895190] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 18:16:22 EDT 2026 [ 340.215802] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 342.183973] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.910028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.911582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.948820] Lustre: Skipped 1 previous similar message [ 359.974912] Lustre: Skipped 3 previous similar messages [ 365.076186] Lustre: server umount lustre-MDT0000 complete [ 369.181398] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683436 with bad export cookie 18126548327894294779 [ 369.194626] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.205692] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 370.145935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.146673] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.160942] Lustre: Skipped 2 previous similar messages [ 370.181226] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 370.193872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.199729] Lustre: Skipped 1 previous similar message [ 374.179366] LustreError: 6550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.179908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.204288] LustreError: 6550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 374.212051] Lustre: Skipped 1 previous similar message [ 375.633278] Lustre: server umount lustre-MDT0001 complete [ 379.311426] Lustre: server umount lustre-OST0000 complete [ 383.545626] Lustre: server umount lustre-OST0001 complete [ 391.071226] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 401.909425] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 445.003401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.306650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.380779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 445.495602] Lustre: lustre-MDT0000: new disk, initializing [ 445.641547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.697693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 451.385558] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 463.037839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.160765] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 463.203349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 463.206729] Lustre: Skipped 1 previous similar message [ 463.260382] Lustre: lustre-MDT0001: new disk, initializing [ 463.366414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.406427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.419488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.336206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 479.920637] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.888422] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.713927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.106566] Lustre: lustre-OST0000: new disk, initializing [ 495.113112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.120812] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.238563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.571553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.586301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.802231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.200858] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 513.770062] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 519.631971] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.853797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.333514] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.605110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 527.842701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 527.860973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.880911] Lustre: Skipped 1 previous similar message [ 527.893705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.449538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.453617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.488583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.824863] Lustre: server umount lustre-OST0000 complete [ 538.081951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.102461] Lustre: Skipped 1 previous similar message [ 543.201430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 543.212420] Lustre: Skipped 4 previous similar messages [ 544.315347] Lustre: server umount lustre-MDT0000 complete [ 547.803673] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683615 with bad export cookie 18126548327894296312 [ 547.808842] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.822519] LustreError: 17055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 548.444271] Lustre: server umount lustre-MDT0001 complete [ 553.184857] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:20:19 (1789683619) [ 555.352879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 557.409386] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:20:24 (1789683624) [ 563.890953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.307275] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.368773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.614347] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 569.831648] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.941518] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.194089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.275580] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 590.171065] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 596.056768] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 603.261925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 603.556268] LustreError: 21286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.393555] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 619.216169] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 629.732067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.735443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 629.746884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 629.758285] Lustre: Skipped 1 previous similar message [ 632.373333] Lustre: server umount lustre-OST0000 complete [ 638.945711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 642.479741] Lustre: server umount lustre-MDT0000 complete [ 645.091120] LustreError: 19461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.114555] LustreError: 19461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 646.242462] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789683713 with bad export cookie 18126548327894297194 [ 646.251884] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 646.254465] LustreError: 19447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 646.628086] Lustre: server umount lustre-MDT0001 complete [ 660.220680] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 663.654160] Key type lgssc unregistered [ 663.945593] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.960954] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.984791] LNet: Removed LNI 192.168.204.107@tcp [ 664.911153] Key type .llcrypt unregistered [ 664.912851] Key type ._llcrypt unregistered [ 688.945757] Key type ._llcrypt registered [ 688.955214] Key type .llcrypt registered [ 689.132797] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 690.489289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.510714] alg: No test for adler32 (adler32-zlib) [ 691.687064] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 691.978850] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 693.704594] Key type lgssc registered [ 694.653710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 703.291443] loop: module loaded [ 703.356950] loop0: detected capacity change from 0 to 204800000 [ 703.547680] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.560754] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.802967] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.903774] loop0: detected capacity change from 0 to 204800000 [ 710.068727] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.078019] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 710.091094] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.652377] loop0: detected capacity change from 0 to 204800000 [ 721.547539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 721.608139] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.991244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 723.028884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 723.167733] Lustre: lustre-MDT0000: new disk, initializing [ 723.279233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.313443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 729.035357] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 737.990062] loop1: detected capacity change from 0 to 204800000 [ 743.899774] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 744.271554] Lustre: lustre-OST07c6: new disk, initializing [ 744.276337] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 744.292319] Lustre: Skipped 1 previous similar message [ 744.302568] Lustre: 26183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 744.394568] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 750.620993] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 754.244684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 754.266082] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 754.316094] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 760.902367] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 760.918151] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 760.923497] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 760.933679] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 760.947442] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 760.955203] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 778.794371] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 778.806948] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 778.825266] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 778.836443] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 778.845122] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 778.856805] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.756834] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 783.769921] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 783.781194] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 783.796240] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.802369] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 783.806340] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.820257] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 783.832253] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.845097] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 783.852802] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.857754] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.863347] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.645060] Lustre: 26467:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 788.662111] Lustre: 26467:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 788.678103] Lustre: 26467:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 788.701373] Lustre: 26467:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.713419] Lustre: 26467:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 788.727975] Lustre: 26467:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.742122] Lustre: 26467:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 788.751947] Lustre: 26467:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.759807] Lustre: 26467:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 788.766364] Lustre: 26467:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 788.779150] Lustre: 26467:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 788.792501] Lustre: 26467:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 796.988436] Lustre: 26467:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 797.010055] Lustre: 26467:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 797.029100] Lustre: 26467:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 797.038213] Lustre: 26467:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.045432] Lustre: 26467:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.052659] Lustre: 26467:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.058334] Lustre: 26467:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 797.063627] Lustre: 26467:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.073543] Lustre: 26467:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 797.078894] Lustre: 26467:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 797.086036] Lustre: 26467:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 797.091977] Lustre: 26467:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.015232] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 808.024147] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 808.035266] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 808.044126] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 808.049842] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.057415] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 808.067460] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 808.074222] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 808.079386] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 808.084716] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 808.091495] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.096788] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 856.549919] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.565556] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.610489] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 861.669579] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 862.641215] Lustre: server umount lustre-OST07c6 complete [ 872.816115] Lustre: server umount lustre-MDT0000 complete [ 890.744303] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 893.801056] Key type lgssc unregistered [ 894.038764] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.048262] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.061351] LNet: Removed LNI 192.168.204.107@tcp [ 894.811519] Key type .llcrypt unregistered [ 894.814845] Key type ._llcrypt unregistered [ 913.487596] Key type ._llcrypt registered [ 913.490402] Key type .llcrypt registered [ 913.602926] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 914.683658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.806673] alg: No test for adler32 (adler32-zlib) [ 915.838273] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 916.143347] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 917.817116] Key type lgssc registered [ 919.165971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 926.308710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 926.333622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.670387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.847626] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 937.961312] LustreError: 30015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.977126] LustreError: 30015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 939.997157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.328935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 944.553135] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 951.340682] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 956.837785] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 963.536657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.987591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.536865] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 973.246346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 976.128377] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.644539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.664159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.665284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.665323] Lustre: Skipped 1 previous similar message [ 988.690235] Lustre: Skipped 1 previous similar message [ 993.761138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 994.542244] Lustre: server umount lustre-OST0000 complete [ 998.885256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.900157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.914044] Lustre: Skipped 2 previous similar messages [ 1004.001728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.007818] Lustre: Skipped 1 previous similar message [ 1004.492197] Lustre: server umount lustre-MDT0000 complete [ 1007.897708] LustreError: 32998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684075 with bad export cookie 13348874364006454483 [ 1007.901618] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.905736] LustreError: 32998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1008.522336] Lustre: server umount lustre-MDT0001 complete [ 1019.874656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1023.738185] Key type lgssc unregistered [ 1024.166805] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.172479] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.204877] LNet: Removed LNI 192.168.204.107@tcp [ 1025.563550] Key type .llcrypt unregistered [ 1025.570419] Key type ._llcrypt unregistered [ 1047.969482] Key type ._llcrypt registered [ 1047.972994] Key type .llcrypt registered [ 1048.093682] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1049.279895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1049.372878] alg: No test for adler32 (adler32-zlib) [ 1050.440449] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1050.687836] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1052.328238] Key type lgssc registered [ 1053.554922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.372417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.391732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.816025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.097957] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1073.126340] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.140895] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1075.889682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.181599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.154689] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1089.178707] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1096.784906] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.563053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.032067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.152248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1111.377822] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1119.266568] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1132.014367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.020206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1132.034119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.013088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.021564] Lustre: Skipped 2 previous similar messages [ 1135.366597] Lustre: server umount lustre-OST0000 complete [ 1142.754940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.764282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.772682] Lustre: Skipped 1 previous similar message [ 1142.777532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.315942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1145.327633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.333050] Lustre: Skipped 1 previous similar message [ 1146.505298] Lustre: server umount lustre-MDT0000 complete [ 1150.434811] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.446143] LustreError: 35016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1150.454991] LustreError: 34996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684217 with bad export cookie 356313697118111677 [ 1150.460873] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.462352] LustreError: 34996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.082769] Lustre: server umount lustre-MDT0001 complete [ 1162.383848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1165.444983] Key type lgssc unregistered [ 1165.877516] LNet: 38550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.888152] LNetError: 38550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.907844] LNet: Removed LNI 192.168.204.107@tcp [ 1166.879077] Key type .llcrypt unregistered [ 1166.881285] Key type ._llcrypt unregistered [ 1187.932549] Key type ._llcrypt registered [ 1187.935923] Key type .llcrypt registered [ 1188.043532] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1188.923072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1188.944345] alg: No test for adler32 (adler32-zlib) [ 1190.031322] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1190.316502] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1192.032415] Key type lgssc registered [ 1193.235867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.253250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1201.286572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.727909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.928266] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1212.905422] LustreError: 39943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.933743] LustreError: 39943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1216.754174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.314465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1222.230550] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1231.568096] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1237.949492] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1245.506277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1245.933212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.000359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1252.832919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1259.634388] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1268.708087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1268.712162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.719525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.626176] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1272.803040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1272.806799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1272.815995] Lustre: Skipped 1 previous similar message [ 1272.837087] Lustre: Skipped 1 previous similar message [ 1274.501846] Lustre: server umount lustre-MDT0000 complete [ 1276.706576] LustreError: 39937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.930290] LustreError: 39943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.956470] LustreError: 39943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1277.966374] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684345 with bad export cookie 424059770793651281 [ 1277.968747] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.010981] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1281.824228] LustreError: 39939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.872058] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 1281.879070] Lustre: Skipped 1 previous similar message [ 1283.043956] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.386286] Lustre: server umount lustre-MDT0001 complete [ 1305.608648] Lustre: server umount lustre-OST0000 complete [ 1323.292523] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1326.600933] Key type lgssc unregistered [ 1326.937723] LNet: 43806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.950631] LNetError: 43806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.979376] LNet: Removed LNI 192.168.204.107@tcp [ 1329.079683] Key type .llcrypt unregistered [ 1329.082089] Key type ._llcrypt unregistered [ 1352.923481] Key type ._llcrypt registered [ 1352.927992] Key type .llcrypt registered [ 1353.072995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1354.465978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.487973] alg: No test for adler32 (adler32-zlib) [ 1355.664600] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1356.026961] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1357.808265] Key type lgssc registered [ 1358.861120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.341432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.378588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.857270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.503685] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1378.281251] LustreError: 45212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.310343] LustreError: 45212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1379.104228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.523510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.654362] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1390.258087] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.303548] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1399.596196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1399.962270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.559950] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1413.086417] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1413.298081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1418.728957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.735662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.756698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1418.775204] Lustre: Skipped 1 previous similar message [ 1420.134299] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1423.843802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1423.857151] Lustre: Skipped 1 previous similar message [ 1428.969176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1428.974820] Lustre: Skipped 2 previous similar messages [ 1433.056538] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1433.348370] Lustre: server umount lustre-OST0000 complete [ 1455.087364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.099679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.111017] Lustre: Skipped 2 previous similar messages [ 1455.140277] Lustre: Skipped 2 previous similar messages [ 1461.489117] Lustre: server umount lustre-MDT0000 complete [ 1464.299719] LustreError: 45940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.326961] LustreError: 45940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1466.803636] LustreError: 45191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684534 with bad export cookie 673413958887948578 [ 1466.816358] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1481.186035] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1481.417517] Lustre: server umount lustre-MDT0001 complete [ 1490.002241] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1492.459906] Key type lgssc unregistered [ 1492.725401] LNet: 48923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.734550] LNetError: 48923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.750726] LNet: Removed LNI 192.168.204.107@tcp [ 1493.543843] Key type .llcrypt unregistered [ 1493.547534] Key type ._llcrypt unregistered [ 1514.572653] Key type ._llcrypt registered [ 1514.575894] Key type .llcrypt registered [ 1514.656891] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1515.515268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.653778] alg: No test for adler32 (adler32-zlib) [ 1516.683408] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1516.906595] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1518.632468] Key type lgssc registered [ 1519.541857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1526.320426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1526.337477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.811657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.759286] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1538.019655] LustreError: 50338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1538.041855] LustreError: 50338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1539.495296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.830274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1543.486398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1550.211730] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1556.232903] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1563.047949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1563.381318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.186757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1573.620079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1575.737782] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.371463] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1584.078040] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1595.850891] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1718h37m42s [ 1597.044928] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1599.459182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.459803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.473401] Lustre: Skipped 2 previous similar messages [ 1599.485315] Lustre: Skipped 1 previous similar message [ 1600.553399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.568092] Lustre: Skipped 2 previous similar messages [ 1603.896960] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1606.629800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.634210] Lustre: Skipped 3 previous similar messages [ 1612.777036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.787551] Lustre: Skipped 2 previous similar messages [ 1613.280435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1613.446855] Lustre: server umount lustre-MDT0000 complete [ 1616.171173] LustreError: 50332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.341365] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1618.925531] LustreError: 50338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.949987] LustreError: 50338:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1621.792227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.871516] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.030409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.071748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1623.083470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1625.230364] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1627.732989] Lustre: DEBUG MARKER: Wait for df (36700) ... [ 1628.734860] Lustre: DEBUG MARKER: done [ 1632.229751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.230260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.240683] LustreError: Skipped 1 previous similar message [ 1632.257440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.267281] Lustre: Skipped 1 previous similar message [ 1637.345637] Lustre: server umount lustre-OST0000 complete [ 1643.489975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.491720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.508777] Lustre: Skipped 2 previous similar messages [ 1650.665782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1650.672238] Lustre: Skipped 7 previous similar messages [ 1654.240315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1654.357345] Lustre: server umount lustre-MDT0000 complete [ 1656.810118] LustreError: 51432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.821158] LustreError: 51432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1657.372962] LustreError: 50317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684724 with bad export cookie 3632793720486728992 [ 1657.375612] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.382149] LustreError: 50317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1657.718582] Lustre: server umount lustre-MDT0001 complete [ 1667.008258] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1669.294752] Key type lgssc unregistered [ 1669.552244] LNet: 55262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.558242] LNetError: 55262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.570938] LNet: Removed LNI 192.168.204.107@tcp [ 1670.285109] Key type .llcrypt unregistered [ 1670.288243] Key type ._llcrypt unregistered [ 1699.165906] Key type ._llcrypt registered [ 1699.167953] Key type .llcrypt registered [ 1699.259372] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1699.815706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.900387] alg: No test for adler32 (adler32-zlib) [ 1700.857043] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1701.002899] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1702.632227] Key type lgssc registered [ 1703.484155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1708.518117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1708.532069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.705671] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1709.721644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1709.838676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.521273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1717.682042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.755916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1717.783354] Lustre: 57450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1717.966351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1720.509150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1725.081676] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1728.470876] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1732.871828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1733.047590] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1733.195944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.608576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1740.370904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1742.651717] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1748.304761] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1718h40m15s [ 1749.265425] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1753.976781] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1755.515832] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1755.554072] LustreError: 59718:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1775.930908] Lustre: lustre-MDT0000: Client 0edccec9-cfed-4c27-8eaa-1aeab07465d9 (at 192.168.204.7@tcp) reconnecting [ 1795.600684] LustreError: 59718:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1797.134774] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1798.619964] Lustre: DEBUG MARKER: done [ 1801.697979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1801.698138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.705730] LustreError: Skipped 1 previous similar message [ 1801.713115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1801.719168] Lustre: Skipped 1 previous similar message [ 1806.822032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1806.825418] Lustre: Skipped 1 previous similar message [ 1807.537298] Lustre: server umount lustre-OST0000 complete [ 1810.402933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1810.414560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.430836] Lustre: Skipped 1 previous similar message [ 1810.434317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1816.196563] Lustre: server umount lustre-MDT0000 complete [ 1818.187067] LustreError: 60213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684885 with bad export cookie 3182304346618260716 [ 1818.188812] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1818.197182] LustreError: 60213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.344382] Lustre: server umount lustre-MDT0001 complete [ 1823.675340] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1825.130829] Key type lgssc unregistered [ 1825.327339] LNet: 61775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.332740] LNetError: 61775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.346487] LNet: Removed LNI 192.168.204.107@tcp [ 1825.791135] Key type .llcrypt unregistered [ 1825.794174] Key type ._llcrypt unregistered [ 1850.368379] Key type ._llcrypt registered [ 1850.370481] Key type .llcrypt registered [ 1850.445777] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1850.977406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.003299] alg: No test for adler32 (adler32-zlib) [ 1851.960441] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 1852.125685] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1853.761855] Key type lgssc registered [ 1854.438727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1856.279142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1856.293589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.456492] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1857.477949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1857.594640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.188633] Lustre: Failing over lustre-MDT0000 [ 1858.316794] Lustre: server umount lustre-MDT0000 complete [ 1861.781835] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:42:09 (1789684929) [ 1865.665598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.957268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.818692] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1874.826556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.903798] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1874.927605] Lustre: 64544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1875.105622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1877.967945] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1882.882255] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1886.357498] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1890.360237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1890.467700] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1890.595763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.137511] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1898.736624] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1899.052919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1903.532980] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1909.218705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.222614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.231155] Lustre: Skipped 1 previous similar message [ 1909.354587] Lustre: server umount lustre-MDT0000 complete [ 1911.511580] LustreError: 64534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789684978 with bad export cookie 8368157872843292992 [ 1911.513590] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.519826] LustreError: 64534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.708509] Lustre: server umount lustre-MDT0001 complete [ 1913.668672] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1916.068519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1920.488223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.504905] LustreError: 63467:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000beaeaf78 x1876620653659136/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1921.686706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.736896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1923.822071] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1927.139459] LustreError: 66388:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1927.140759] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.147920] LustreError: 66388:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1927.158696] Lustre: Skipped 1 previous similar message [ 1927.821476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1929.128550] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1930.144288] Lustre: 63468:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789684981/real 1789684981] req@0000000085b40169 x1876620653658752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789684997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1930.352108] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1932.192310] Lustre: 63470:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789684983/real 1789684983] req@00000000a29e51f3 x1876620653658880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789684999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1934.610377] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.332729] Lustre: 63470:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789684986/real 1789684986] req@00000000f69a94e0 x1876620653659008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789685002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1937.823165] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.521901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1943.528252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.542271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.546129] Lustre: Skipped 2 previous similar messages [ 1944.547695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1944.551498] Lustre: Skipped 2 previous similar messages [ 1947.203688] Lustre: server umount lustre-MDT0000 complete [ 1949.256196] LustreError: 66397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685016 with bad export cookie 8368157872843295267 [ 1949.261633] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.264932] LustreError: 66397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1949.422581] Lustre: server umount lustre-MDT0001 complete [ 1953.550215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1959.859795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.984169] Lustre: 63469:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685017/real 1789685017] req@000000002ce2a03d x1876620653681280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789685033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1966.009573] Lustre: 63469:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1966.016981] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.024372] Lustre: Skipped 2 previous similar messages [ 1971.169193] Lustre: 63469:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685022/real 1789685022] req@00000000745d25ed x1876620653681408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789685038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.242624] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7421ab3fbb368f86 [ 1974.251505] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 1974.254860] Lustre: Skipped 1 previous similar message [ 1974.426053] LustreError: 66389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.470802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.475649] Lustre: Skipped 1 previous similar message [ 1974.507812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1976.547539] LustreError: 70989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.557854] LustreError: 70989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1976.773327] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1979.616666] Lustre: 63471:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685031/real 1789685031] req@00000000bed46f0f x1876620653681792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789685047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1979.632943] Lustre: 63471:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1981.709833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.387335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1985.065387] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1988.932807] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1992.490643] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.283378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.291551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.310236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.697092] Lustre: server umount lustre-MDT0000 complete [ 2004.849151] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685072 with bad export cookie 8368157872843296646 [ 2004.852118] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2004.863941] LustreError: 64535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2005.072271] Lustre: server umount lustre-MDT0001 complete [ 2009.090901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2011.061670] Lustre: DEBUG MARKER: files compared the same [ 2022.483770] Lustre: server umount lustre-OST0000 complete [ 2033.069025] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2034.852533] Key type lgssc unregistered [ 2035.080051] LNet: 74257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.087961] LNetError: 74257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.101494] LNet: Removed LNI 192.168.204.107@tcp [ 2035.662709] Key type .llcrypt unregistered [ 2035.666374] Key type ._llcrypt unregistered [ 2049.279498] Key type ._llcrypt registered [ 2049.281582] Key type .llcrypt registered [ 2049.347327] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2049.896888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.930443] alg: No test for adler32 (adler32-zlib) [ 2050.911717] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2051.063364] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2052.688229] Key type lgssc registered [ 2053.462243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.498201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2059.523543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.958206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.408490] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2070.219943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.442438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.144216] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2078.012779] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2081.040688] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2084.845466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2085.050235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.057257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2088.533447] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2094.511035] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2101.728934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2101.729553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.734502] LustreError: Skipped 1 previous similar message [ 2101.735516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2101.743099] Lustre: Skipped 1 previous similar message [ 2101.756489] Lustre: Skipped 1 previous similar message [ 2102.971773] Lustre: server umount lustre-OST0000 complete [ 2106.338574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2106.343656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.349820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2111.671673] Lustre: server umount lustre-MDT0000 complete [ 2111.971272] LustreError: 75642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2111.994614] LustreError: 75642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2113.961272] LustreError: 75628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685181 with bad export cookie 7639941498817973663 [ 2113.966461] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2113.967600] LustreError: 75628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2114.147425] Lustre: server umount lustre-MDT0001 complete [ 2120.740916] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2122.439168] Key type lgssc unregistered [ 2122.614826] LNet: 79175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.618792] LNetError: 79175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.689481] LNet: Removed LNI 192.168.204.107@tcp [ 2124.249907] Key type .llcrypt unregistered [ 2124.253363] Key type ._llcrypt unregistered [ 2139.066681] Key type ._llcrypt registered [ 2139.068797] Key type .llcrypt registered [ 2139.136536] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2139.793245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.833153] alg: No test for adler32 (adler32-zlib) [ 2140.823251] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2140.981051] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2142.616635] Key type lgssc registered [ 2143.435847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2148.564978] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2148.577260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2164.073912] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2169.184307] LustreError: 80539:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 2196.000519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.740348] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2210.672435] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2218.426804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.611645] Lustre: *** cfs_fail_loc=706, val=0*** [ 2218.865167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.918393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2223.681730] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2224.100477] LustreError: 80564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.107352] LustreError: 80564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2229.227203] LustreError: 80565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.283059] LustreError: 80564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.291024] LustreError: 80564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2232.207034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.474925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.109724] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2244.339244] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.496812] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.287253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.287622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.297301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.310092] Lustre: Skipped 1 previous similar message [ 2258.401551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.416061] Lustre: Skipped 2 previous similar messages [ 2258.654992] Lustre: server umount lustre-OST0000 complete [ 2263.010103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.018575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.030781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.540210] Lustre: server umount lustre-MDT0000 complete [ 2268.645766] LustreError: 81631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.660357] LustreError: 81631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2271.421110] LustreError: 80545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685338 with bad export cookie 14731884035780886848 [ 2271.426820] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.436640] LustreError: 80545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2271.669067] Lustre: server umount lustre-MDT0001 complete [ 2280.878730] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2283.146497] Key type lgssc unregistered [ 2283.369042] LNet: 84155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.374843] LNetError: 84155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.389093] LNet: Removed LNI 192.168.204.107@tcp [ 2284.122342] Key type .llcrypt unregistered [ 2284.124735] Key type ._llcrypt unregistered [ 2304.498566] Key type ._llcrypt registered [ 2304.500897] Key type .llcrypt registered [ 2304.598855] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2305.301564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.324080] alg: No test for adler32 (adler32-zlib) [ 2306.352383] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2306.615869] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2308.264341] Key type lgssc registered [ 2309.184734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2315.560245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2315.587720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.850966] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2326.912913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.346118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.130908] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2332.641844] LustreError: 86233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.660409] LustreError: 86233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2336.740479] LustreError: 86237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.761660] LustreError: 86237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2338.895780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.178092] LustreError: 86232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.204470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.207353] LustreError: 86232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2344.233806] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2344.426384] LustreError: 86236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.454396] LustreError: 86236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2349.539939] LustreError: 86237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.559719] LustreError: 86237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2351.234617] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2351.243403] Lustre: MGS: Client 3836bc44-9b13-4728-b8d0-f1606d6b949d (at 0@lo) reconnecting [ 2351.258579] LustreError: 86247:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2351.436146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2351.495441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2354.955147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2367.969620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2367.981878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.987152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.999803] Lustre: Skipped 1 previous similar message [ 2373.091614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2373.099940] Lustre: Skipped 1 previous similar message [ 2373.386660] Lustre: server umount lustre-OST0000 complete [ 2378.213928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.217608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.233269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.246242] Lustre: Skipped 2 previous similar messages [ 2382.578829] Lustre: server umount lustre-MDT0000 complete [ 2383.330196] LustreError: 86236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.358241] LustreError: 86236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2385.759720] Lustre: server umount lustre-MDT0001 complete [ 2388.819547] Lustre: server umount MGS complete [ 2399.835902] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 18:51:06 (1789685466) [ 2425.480863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 2433.960685] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2460.249311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.640330] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2470.741782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.909731] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2470.941280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2470.994308] Lustre: lustre-MDT0001: new disk, initializing [ 2471.039680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2471.057971] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2471.062905] Lustre: 93398:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2471.068302] LustreError: 93398:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2473.757474] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2481.018671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.206677] Lustre: lustre-OST0000: new disk, initializing [ 2481.211592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2481.219984] Lustre: 94163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2481.273543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2486.498233] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2487.844368] Lustre: 94537:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d07b27b5 x1876621130766848/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789685571 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2493.833623] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2498.881742] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.897248] Lustre: MGS: Client 14d2d520-5328-4a36-b30e-06e418873f94 (at 0@lo) reconnecting [ 2498.924719] Lustre: 92666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2498.939554] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2498.945586] LustreError: 93384:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2498.953491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2499.027465] Lustre: lustre-MDT0000: new disk, initializing [ 2499.127145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2499.144064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2499.219796] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2500.194458] Lustre: 95232:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000590cc58a x1876621130773376/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789685583 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2501.753900] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2508.275757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2508.281221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2509.287788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2509.292593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2509.353960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2529.762042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2529.773897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2529.784573] Lustre: Skipped 1 previous similar message [ 2529.789028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2532.844794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.588586] Lustre: server umount lustre-OST0000 complete [ 2543.073571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.081048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.097629] Lustre: Skipped 1 previous similar message [ 2543.122450] Lustre: Skipped 3 previous similar messages [ 2553.316597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.328231] Lustre: Skipped 3 previous similar messages [ 2555.360183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2555.639385] Lustre: server umount lustre-MDT0000 complete [ 2558.440637] LustreError: 93373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.502740] LustreError: 93373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2577.376475] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2577.950801] Lustre: server umount lustre-MDT0001 complete [ 2583.335946] Lustre: server umount MGS complete [ 2600.231737] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 18:54:25 (1789685665) [ 2629.401774] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2632.724756] Key type lgssc unregistered [ 2633.033878] LNet: 97751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.044731] LNetError: 97751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.080209] LNet: Removed LNI 192.168.204.107@tcp [ 2633.937532] Key type .llcrypt unregistered [ 2633.939397] Key type ._llcrypt unregistered [ 2654.579857] Key type ._llcrypt registered [ 2654.585444] Key type .llcrypt registered [ 2654.760751] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2656.487082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2656.543943] alg: No test for adler32 (adler32-zlib) [ 2657.588063] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2657.964515] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2659.632371] Key type lgssc registered [ 2660.917157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2670.975820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.036196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.578686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.928766] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2681.830767] LustreError: 98971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.863104] LustreError: 98971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2686.946072] LustreError: 98972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.933898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.480834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2693.406220] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2701.672105] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2707.313094] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2713.373521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2713.684753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.785490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2720.150408] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2727.980140] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2741.428206] Lustre: server umount lustre-OST0000 complete [ 2748.899071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.911724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2748.918983] Lustre: Skipped 1 previous similar message [ 2754.017947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.022034] Lustre: Skipped 2 previous similar messages [ 2756.325580] Lustre: server umount lustre-MDT0000 complete [ 2759.148519] LustreError: 98972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.169256] LustreError: 98972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2760.161527] Lustre: 98283:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789685811/real 1789685811] req@00000000a644bb34 x1876621498805888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789685827 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2760.176767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.475103] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685827 with bad export cookie 15219441795594842744 [ 2760.476682] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2760.489188] LustreError: 98952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2760.974893] Lustre: server umount lustre-MDT0001 complete [ 2772.115240] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2775.626150] Key type lgssc unregistered [ 2775.878709] LNet: 102503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.896352] LNetError: 102503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.918344] LNet: Removed LNI 192.168.204.107@tcp [ 2776.910237] Key type .llcrypt unregistered [ 2776.916442] Key type ._llcrypt unregistered [ 2794.358178] Key type ._llcrypt registered [ 2794.367376] Key type .llcrypt registered [ 2794.582419] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2795.675775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2795.777198] alg: No test for adler32 (adler32-zlib) [ 2796.899264] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 2797.107224] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2798.776292] Key type lgssc registered [ 2800.070190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2806.404355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2806.410918] LustreError: 103578:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2806.441364] LustreError: 103583:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2806.446495] LustreError: 103578:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 2806.454245] LustreError: 103578:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2806.460699] LustreError: 103578:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2806.468897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.901982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.515721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.779492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2821.441346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2827.052408] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lsmod [ 2830.293293] LustreError: 104777:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2830.454843] LustreError: 104789:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2830.466717] LustreError: 104777:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4844 [ 2830.479584] LustreError: 104777:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2830.493416] LustreError: 104777:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2830.509725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.966675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.014639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2837.476639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2837.481857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.500064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2838.497850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2838.503418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.512060] Lustre: Skipped 1 previous similar message [ 2838.529027] Lustre: Skipped 1 previous similar message [ 2840.331682] Lustre: server umount lustre-MDT0000 complete [ 2841.639336] LustreError: 103616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.667576] LustreError: 103616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2844.029175] LustreError: 103595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789685911 with bad export cookie 18052042205546193939 [ 2844.042528] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2845.792378] LustreError: 104051:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000da340a55 x1876621644551296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2846.030250] Lustre: server umount lustre-MDT0001 complete [ 2860.770958] Lustre: server umount lustre-OST0000 complete [ 2868.978704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.350257] LustreError: 105855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.410455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.540932] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2874.861716] LustreError: 105856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.972646] LustreError: 105855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2884.068240] LustreError: 105856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.557720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.067965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2891.774429] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2899.578761] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2906.032688] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2913.050598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.411457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2914.431706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 2919.404755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2926.983680] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.858192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2938.863181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.900764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.912093] Lustre: Skipped 1 previous similar message [ 2941.413767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.414347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.415745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.437331] Lustre: Skipped 1 previous similar message [ 2944.339498] Lustre: server umount lustre-OST0000 complete [ 2951.658898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.676684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.680800] Lustre: Skipped 1 previous similar message [ 2951.706684] Lustre: Skipped 1 previous similar message [ 2960.871252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.883102] Lustre: Skipped 3 previous similar messages [ 2962.401313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.693758] Lustre: server umount lustre-MDT0000 complete [ 2965.994383] LustreError: 105856:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.027200] LustreError: 105856:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2966.634869] LustreError: 105837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686034 with bad export cookie 18052042205546194870 [ 2966.636592] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.652392] LustreError: 105837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2967.201305] Lustre: server umount lustre-MDT0001 complete [ 2978.465607] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2980.968618] Key type lgssc unregistered [ 2981.264664] LNet: 109391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.270358] LNetError: 109391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.286700] LNet: Removed LNI 192.168.204.107@tcp [ 2982.087296] Key type .llcrypt unregistered [ 2982.089286] Key type ._llcrypt unregistered [ 3006.991524] Key type ._llcrypt registered [ 3006.994659] Key type .llcrypt registered [ 3007.156928] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3008.185627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3008.282881] alg: No test for adler32 (adler32-zlib) [ 3009.274847] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3009.488949] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3011.157874] Key type lgssc registered [ 3012.517218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3021.619814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3021.639764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.321411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.511385] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3033.577987] LustreError: 110819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.620819] LustreError: 110819:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3038.698117] LustreError: 110820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.812530] LustreError: 110819:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.493459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.901853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.750271] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3058.378698] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3066.721567] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3076.445597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3077.051672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3085.005211] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3093.121349] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3093.509862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3099.077352] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3103.608617] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3108.845835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.853886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.858549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.865015] Lustre: Skipped 1 previous similar message [ 3108.874087] Lustre: Skipped 1 previous similar message [ 3113.969857] LustreError: 113313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.216377] Lustre: server umount lustre-OST0000 complete [ 3121.636440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3121.649781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.671237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.082445] Lustre: server umount lustre-MDT0000 complete [ 3127.267679] LustreError: 111706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.315246] LustreError: 111706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3133.965837] LustreError: 110801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686201 with bad export cookie 14592054052647992416 [ 3133.976104] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.979662] LustreError: 110801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3134.044868] LustreError: 110129:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 3135.857766] Lustre: server umount lustre-MDT0001 complete [ 3153.406946] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3157.109486] Key type lgssc unregistered [ 3157.501229] LNet: 114483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.515380] LNetError: 114483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.535815] LNet: Removed LNI 192.168.204.107@tcp [ 3158.778568] Key type .llcrypt unregistered [ 3158.786780] Key type ._llcrypt unregistered [ 3173.545337] Key type ._llcrypt registered [ 3173.547868] Key type .llcrypt registered [ 3173.647391] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3174.312847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3174.335232] alg: No test for adler32 (adler32-zlib) [ 3175.446206] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3175.672397] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3177.344393] Key type lgssc registered [ 3178.553431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3185.404141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3185.414145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.882990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.601320] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3197.417578] LustreError: 115680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.451631] LustreError: 115680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3197.609867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.886043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.643775] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3208.115703] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3213.107720] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3218.507343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3218.847638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.238502] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3229.964810] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3232.185529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3242.471772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3242.474887] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3242.483543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3242.495802] Lustre: Skipped 1 previous similar message [ 3246.727565] Lustre: server umount lustre-OST0000 complete [ 3254.249738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3254.264873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.332557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.350836] Lustre: Skipped 1 previous similar message [ 3257.839854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.843942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3260.166850] Lustre: server umount lustre-MDT0000 complete [ 3262.058313] LustreError: 115675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.116021] LustreError: 115675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3267.556257] LustreError: 115681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.610953] LustreError: 115681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3269.161224] LustreError: 115661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686336 with bad export cookie 5297375037835501000 [ 3269.179320] LustreError: 115661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3269.183453] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3270.166146] Lustre: server umount lustre-MDT0001 complete [ 3290.271249] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3293.413865] Key type lgssc unregistered [ 3293.729976] LNet: 119227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3293.741733] LNetError: 119227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3293.767204] LNet: Removed LNI 192.168.204.107@tcp [ 3294.579396] Key type .llcrypt unregistered [ 3294.582768] Key type ._llcrypt unregistered [ 3322.625945] Key type ._llcrypt registered [ 3322.648826] Key type .llcrypt registered [ 3322.825500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3324.357715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3324.813512] alg: No test for adler32 (adler32-zlib) [ 3326.038255] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3326.361058] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3328.032343] Key type lgssc registered [ 3328.918336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3337.001651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3337.037188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.387753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.952329] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3347.940318] LustreError: 120644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.967021] LustreError: 120644:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3351.297641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.715062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.896366] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3364.251452] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.679518] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3377.601340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.045632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.183149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3384.485344] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3392.558578] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3404.074491] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3409.182829] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3409.190734] Lustre: Skipped 2 previous similar messages [ 3411.544360] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3411.600599] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3415.257518] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3415.263025] Lustre: Skipped 2 previous similar messages [ 3418.603622] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3418.644642] Lustre: Skipped 1 previous similar message [ 3424.282987] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3424.291455] Lustre: Skipped 2 previous similar messages [ 3428.251460] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp [ 3434.838219] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.7/24@tcp in log lustre-MDT0000 [ 3434.845246] Lustre: Skipped 2 previous similar messages [ 3438.342945] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3438.360436] Lustre: Skipped 1 previous similar message [ 3448.811171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3448.812070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.832538] LustreError: Skipped 1 previous similar message [ 3448.843231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3448.854876] Lustre: Skipped 1 previous similar message [ 3453.685488] Lustre: server umount lustre-OST0000 complete [ 3459.044156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3459.045204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.073250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.086785] Lustre: Skipped 1 previous similar message [ 3459.111932] Lustre: Skipped 2 previous similar messages [ 3464.174156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.176454] Lustre: Skipped 1 previous similar message [ 3465.040168] Lustre: server umount lustre-MDT0000 complete [ 3469.281817] LustreError: 120645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.316577] LustreError: 120645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3469.602675] LustreError: 120625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686537 with bad export cookie 18430955449360130066 [ 3469.608355] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.612052] LustreError: 120625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3469.918473] Lustre: server umount lustre-MDT0001 complete [ 3480.571570] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3483.103621] Key type lgssc unregistered [ 3483.396804] LNet: 125446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3483.406691] LNetError: 125446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3483.421417] LNet: Removed LNI 192.168.204.107@tcp [ 3484.300824] Key type .llcrypt unregistered [ 3484.302649] Key type ._llcrypt unregistered [ 3509.671653] Key type ._llcrypt registered [ 3509.674267] Key type .llcrypt registered [ 3509.793451] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3510.678649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3510.788575] alg: No test for adler32 (adler32-zlib) [ 3511.858106] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3512.151520] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3513.872353] Key type lgssc registered [ 3515.467929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3522.103650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3522.124737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.510750] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3523.525940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.532245] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.7/24@tcp [ 3527.241954] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3532.776857] LustreError: 127003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3532.797720] LustreError: 127003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3534.081695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.402536] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3534.423785] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3534.482187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3538.010159] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3543.517281] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3547.493967] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3552.174534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3552.427832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.583429] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3561.784491] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3567.793469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3575.265683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.275354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.284095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.344351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.345931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.347552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.357356] Lustre: Skipped 1 previous similar message [ 3580.491857] Lustre: server umount lustre-OST0000 complete [ 3583.461517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.462588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3583.479053] Lustre: Skipped 1 previous similar message [ 3583.494312] Lustre: Skipped 1 previous similar message [ 3588.578087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.585117] Lustre: Skipped 1 previous similar message [ 3589.561114] Lustre: server umount lustre-MDT0000 complete [ 3592.740443] LustreError: 126985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686660 with bad export cookie 4187763066448727181 [ 3592.747444] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.750250] LustreError: 126985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3592.993725] Lustre: server umount lustre-MDT0001 complete [ 3601.345995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3603.517342] Key type lgssc unregistered [ 3603.821702] LNet: 130587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3603.827923] LNetError: 130587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3603.855861] LNet: Removed LNI 192.168.204.107@tcp [ 3604.685823] Key type .llcrypt unregistered [ 3604.689586] Key type ._llcrypt unregistered [ 3624.297630] Key type ._llcrypt registered [ 3624.300262] Key type .llcrypt registered [ 3624.381349] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3633.859588] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3634.546743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3634.560674] alg: No test for adler32 (adler32-zlib) [ 3635.523068] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3635.703350] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3637.336683] Key type lgssc registered [ 3638.074589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3662.004255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3662.033688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.160955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3663.228601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3663.293698] Lustre: lustre-MDT0000: new disk, initializing [ 3663.359293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.375994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3665.909689] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3673.396090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.477924] Lustre: 133960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3673.528872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3673.535825] Lustre: Skipped 1 previous similar message [ 3673.615144] Lustre: lustre-MDT0001: new disk, initializing [ 3673.683401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3673.725868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3673.733922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3676.551076] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3683.329736] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3687.069629] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3691.872219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3692.052605] Lustre: lustre-OST0000: new disk, initializing [ 3692.055598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3692.059758] Lustre: 135927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3692.120839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.776616] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3700.266519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3700.279577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3700.355965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3701.607663] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.278414] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3704.420815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3706.981662] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3707.125959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3710.435324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.444867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.458259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3710.949364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3710.951205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.967636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3714.556789] Lustre: server umount lustre-OST0000 complete [ 3719.653946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3719.657302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.668632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.195033] Lustre: server umount lustre-MDT0000 complete [ 3725.445163] LustreError: 133951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686792 with bad export cookie 16598567826955280862 [ 3725.448110] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3725.455720] LustreError: 133951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3725.646321] Lustre: server umount lustre-MDT0001 complete [ 3730.670867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.969176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.369697] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3736.033747] LustreError: 138141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.045660] LustreError: 138141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3738.323440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.588542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3741.411759] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3745.735947] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.136926] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3753.430471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.660419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.811335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3762.630113] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3765.546334] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3765.712444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3768.721977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3768.890149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3775.804936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.943661] Lustre: lustre-OST0001: new disk, initializing [ 3775.944277] Lustre: lustre-OST0001: Not available for connect from 192.168.204.7@tcp (not set up) [ 3775.948979] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3775.954274] Lustre: Skipped 2 previous similar messages [ 3775.966086] Lustre: 141508:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3776.024779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3779.960210] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3781.664834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3781.672667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3781.721586] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3785.646421] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3785.786871] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3788.648480] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3788.799679] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3793.120894] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b9502f14000.ost_server_uuid 50 [ 3794.338376] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b9502f14000.ost_server_uuid in FULL state after 0 sec [ 3802.089202] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.090635] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.100429] LustreError: Skipped 1 previous similar message [ 3802.101316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3802.120677] Lustre: Skipped 2 previous similar messages [ 3805.388893] Lustre: server umount lustre-OST0001 complete [ 3807.212843] LustreError: 139961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3807.228574] LustreError: 139961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3809.253901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3809.266618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.331982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3812.334312] LustreError: 141509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.339617] Lustre: Skipped 2 previous similar messages [ 3812.350507] LustreError: 141509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3814.133158] Lustre: server umount lustre-OST0000 complete [ 3817.443114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.462617] Lustre: Skipped 2 previous similar messages [ 3831.264251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3831.430210] Lustre: server umount lustre-MDT0000 complete [ 3832.804758] LustreError: 138142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3832.815060] LustreError: 138142:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3834.061938] LustreError: 141507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789686901 with bad export cookie 16598567826955281737 [ 3834.065459] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.069901] LustreError: 141507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3834.290451] Lustre: server umount lustre-MDT0001 complete [ 3847.555735] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3849.492529] Key type lgssc unregistered [ 3849.702584] LNet: 144510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3849.710605] LNetError: 144510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3849.721690] LNet: Removed LNI 192.168.204.107@tcp [ 3850.324871] Key type .llcrypt unregistered [ 3850.327325] Key type ._llcrypt unregistered [ 3878.458173] Key type ._llcrypt registered [ 3878.460507] Key type .llcrypt registered [ 3878.543880] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3888.504281] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3889.241202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3889.255844] alg: No test for adler32 (adler32-zlib) [ 3890.218934] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 3890.381966] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3891.992294] Key type lgssc registered [ 3892.782550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.970359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.990361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.178417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3916.198179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3916.251795] Lustre: lustre-MDT0000: new disk, initializing [ 3916.335310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.347669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3919.098568] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3926.590975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3926.677697] Lustre: 148638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3926.729129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3926.733471] Lustre: Skipped 1 previous similar message [ 3926.805534] Lustre: lustre-MDT0001: new disk, initializing [ 3926.872521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.901335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3926.912785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3929.710326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3935.791521] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3939.299776] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.755909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.956779] Lustre: lustre-OST0000: new disk, initializing [ 3943.959927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3943.965512] Lustre: 150606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3944.035726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.234408] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3953.695145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3953.703498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3953.765589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3954.170402] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3956.938892] Lustre: Failing over lustre-OST0000 [ 3957.050402] Lustre: server umount lustre-OST0000 complete [ 3971.804238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.001065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.009169] LustreError: 146861:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 3972.021127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3972.033295] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3975.072111] Lustre: 146862:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687026/real 1789687026] req@00000000afe27065 x1876622790652288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789687042 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3975.559346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3975.998179] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3976.160160] Lustre: 146861:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687027/real 1789687027] req@000000004411b063 x1876622790652800/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789687043 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3976.172888] Lustre: 146861:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3977.209944] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3977.210131] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 3977.813199] Lustre: Failing over lustre-MDT0000 [ 3977.976201] Lustre: server umount lustre-MDT0000 complete [ 3978.209192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3978.220049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.231460] Lustre: Skipped 1 previous similar message [ 3979.872530] Lustre: 146862:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687031/real 1789687031] req@0000000080ccd9ea x1876622790653824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789687047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3982.307212] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.376563] Lustre: 146863:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789687036/real 1789687036] req@00000000ac006e5b x1876622790654976/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789687052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3986.206901] LustreError: 148643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.225976] LustreError: 148643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3987.427607] LustreError: 148645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.451254] LustreError: 148645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3991.331661] LustreError: 148644:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.177888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.306425] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3994.316368] Lustre: 148641:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3994.495812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.537939] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3996.453678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3997.656226] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3999.725155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3999.730509] Lustre: Skipped 1 previous similar message [ 3999.749926] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3999.782964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4004.836656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.839302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.846295] Lustre: Skipped 2 previous similar messages [ 4004.853996] Lustre: Skipped 1 previous similar message [ 4009.084845] Lustre: server umount lustre-OST0000 complete [ 4015.073933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.075541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4015.077346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.092127] Lustre: Skipped 1 previous similar message [ 4017.618111] Lustre: server umount lustre-MDT0000 complete [ 4019.872333] LustreError: 148628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687087 with bad export cookie 11093791777955252377 [ 4019.874600] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4019.881133] LustreError: 148628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4020.126776] Lustre: server umount lustre-MDT0001 complete [ 4026.717973] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4028.537900] Key type lgssc unregistered [ 4028.737874] LNet: 154257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4028.745598] LNetError: 154257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4028.760724] LNet: Removed LNI 192.168.204.107@tcp [ 4029.301736] Key type .llcrypt unregistered [ 4029.303251] Key type ._llcrypt unregistered [ 4056.341773] Key type ._llcrypt registered [ 4056.343780] Key type .llcrypt registered [ 4056.415141] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4057.042562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4057.122716] alg: No test for adler32 (adler32-zlib) [ 4058.153143] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 4058.312400] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4059.944978] Key type lgssc registered [ 4060.758753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4066.232594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4066.249147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.547895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.392176] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4077.032972] LustreError: 155772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.047396] LustreError: 155772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4077.063859] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4079.939761] LustreError: 155801:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4079.947255] LustreError: 155801:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4079.952907] LustreError: 155801:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 4080.189178] Lustre: server umount lustre-MDT0000 complete [ 4089.281982] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4096.523358] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4159.290836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.702612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4159.742695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4159.856508] Lustre: lustre-MDT0000: new disk, initializing [ 4159.974614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.003610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4166.540319] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4180.024399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.274621] Lustre: 159107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4180.338870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4180.343708] Lustre: Skipped 1 previous similar message [ 4180.480628] Lustre: lustre-MDT0001: new disk, initializing [ 4180.594681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.651807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4180.675057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4186.801972] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4198.158800] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4204.637502] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4211.709533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4212.001210] Lustre: lustre-OST0000: new disk, initializing [ 4212.006979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4212.015701] Lustre: 161078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4212.125260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4218.124520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4219.996341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4220.006311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4220.105867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4227.501746] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4232.323480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4233.970323] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:21:40 (1789687300) [ 4507.106841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.110885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.136750] Lustre: Skipped 1 previous similar message [ 4507.151647] Lustre: Skipped 2 previous similar messages [ 4510.568508] Lustre: server umount lustre-MDT0000 complete [ 4512.224673] LustreError: 159118:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.243348] LustreError: 159118:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4514.571815] LustreError: 159099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687581 with bad export cookie 17540111385596687778 [ 4514.572242] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4514.580931] LustreError: 159099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4517.345389] LustreError: 159115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.345987] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.390611] Lustre: Skipped 1 previous similar message [ 4517.403694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4519.392722] LustreError: 162854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4519.393353] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4520.925265] Lustre: server umount lustre-MDT0001 complete [ 4530.828307] Lustre: server umount lustre-OST0000 complete [ 4545.442795] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4559.347820] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4607.593722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.019926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4608.051492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4608.220404] Lustre: lustre-MDT0000: new disk, initializing [ 4608.313543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.343545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4613.249130] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4625.070040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.304446] Lustre: 165744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4625.413267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4625.426746] Lustre: Skipped 1 previous similar message [ 4625.598545] Lustre: lustre-MDT0001: new disk, initializing [ 4625.773909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4625.861048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4625.891169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4632.089583] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4643.545772] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4650.631499] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4657.803850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4658.225923] Lustre: lustre-OST0000: new disk, initializing [ 4658.230757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4658.241256] Lustre: 167711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4658.385490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.729256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4666.948952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4666.970850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4667.139428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4676.495748] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.711318] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4683.014082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4688.637326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4688.856471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4692.961561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4692.964227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.974386] LustreError: Skipped 1 previous similar message [ 4692.975694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4693.020576] Lustre: Skipped 1 previous similar message [ 4697.461267] Lustre: server umount lustre-OST0000 complete [ 4702.706029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4702.723556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.735346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4702.746027] Lustre: Skipped 1 previous similar message [ 4708.124655] Lustre: server umount lustre-MDT0000 complete [ 4708.327028] LustreError: 165751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.358957] LustreError: 165751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4712.234639] LustreError: 165736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687779 with bad export cookie 17540111385596846608 [ 4712.237881] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.241165] LustreError: 165736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4712.625848] Lustre: server umount lustre-MDT0001 complete [ 4722.658129] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:29:48 (1789687788) [ 4743.975900] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4755.981538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4796.818546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.138474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4797.191695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4797.292766] Lustre: lustre-MDT0000: new disk, initializing [ 4797.373077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4797.406388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4802.320391] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4813.049691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.207238] Lustre: 172478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4813.572291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4818.669328] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4831.143127] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4837.848627] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4845.326795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.543675] Lustre: lustre-OST0000: new disk, initializing [ 4845.549510] Lustre: Skipped 1 previous similar message [ 4845.553328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.556538] Lustre: Skipped 2 previous similar messages [ 4845.560641] Lustre: 174458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4852.632267] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4854.352125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4854.364726] Lustre: Skipped 1 previous similar message [ 4854.376723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4854.501582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4863.772743] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4873.198572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4873.201941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4873.215509] Lustre: Skipped 2 previous similar messages [ 4873.228562] Lustre: Skipped 3 previous similar messages [ 4877.614859] Lustre: server umount lustre-OST0000 complete [ 4883.438595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.461126] Lustre: Skipped 1 previous similar message [ 4888.335977] Lustre: server umount lustre-MDT0000 complete [ 4888.554553] LustreError: 172490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.576376] LustreError: 172490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4893.736790] LustreError: 172472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789687961 with bad export cookie 17540111385596847490 [ 4893.774200] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.248662] Lustre: server umount lustre-MDT0001 complete [ 4906.668466] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:32:52 (1789687972) [ 4925.913333] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4937.043079] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4978.370274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.698364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4978.754896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4978.878735] Lustre: lustre-MDT0000: new disk, initializing [ 4978.972931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.976974] Lustre: Skipped 2 previous similar messages [ 4978.998873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4983.818183] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4994.729199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4994.848437] Lustre: 178782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4995.054774] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5000.212986] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5011.644661] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5017.956824] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5025.652898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.111325] Lustre: 180761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5033.396105] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5033.541434] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5033.692333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5044.749847] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5054.441663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5054.443092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.461574] LustreError: Skipped 1 previous similar message [ 5054.464383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5054.484086] Lustre: Skipped 1 previous similar message [ 5054.510654] Lustre: Skipped 3 previous similar messages [ 5059.567786] LustreError: 180743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.593230] LustreError: 180743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5059.693923] Lustre: server umount lustre-OST0000 complete [ 5070.034636] Lustre: server umount lustre-MDT0000 complete [ 5074.178823] LustreError: 178774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688141 with bad export cookie 17540111385596848575 [ 5074.181433] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5074.189158] LustreError: 178774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5074.730630] Lustre: server umount lustre-MDT0001 complete [ 5085.479507] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5089.224940] Key type lgssc unregistered [ 5089.785902] LNet: 182617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5089.817790] LNetError: 182617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5089.841498] LNet: Removed LNI 192.168.204.107@tcp [ 5091.388795] Key type .llcrypt unregistered [ 5091.391494] Key type ._llcrypt unregistered [ 5123.951938] Key type ._llcrypt registered [ 5123.954540] Key type .llcrypt registered [ 5124.045977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5125.083409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5125.164669] alg: No test for adler32 (adler32-zlib) [ 5126.263790] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5126.583304] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5128.354648] Key type lgssc registered [ 5129.682817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5138.868392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5138.900560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.305601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5145.241246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5149.674414] LustreError: 184034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.687351] LustreError: 184034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5154.792076] LustreError: 184035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.857968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.529411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.333230] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5169.693672] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5175.686246] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5183.090557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.573920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.635437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5190.780886] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5199.817843] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5220.834548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5220.847520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.863466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.857779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5221.866510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.880948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.901971] Lustre: Skipped 1 previous similar message [ 5226.074138] Lustre: server umount lustre-OST0000 complete [ 5232.100940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5232.104340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5232.117734] Lustre: Skipped 1 previous similar message [ 5232.135721] Lustre: Skipped 1 previous similar message [ 5236.973040] Lustre: server umount lustre-MDT0000 complete [ 5237.221170] LustreError: 184763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.245263] LustreError: 184763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5242.252619] LustreError: 184014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688309 with bad export cookie 6830551332116168759 [ 5242.273218] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5242.276499] LustreError: 184014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5242.965835] Lustre: server umount lustre-MDT0001 complete [ 5257.777493] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5261.234545] Key type lgssc unregistered [ 5261.686506] LNet: 187591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5261.710373] LNetError: 187591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5261.739048] LNet: Removed LNI 192.168.204.107@tcp [ 5262.987472] Key type .llcrypt unregistered [ 5262.989240] Key type ._llcrypt unregistered [ 5291.918821] Key type ._llcrypt registered [ 5291.920513] Key type .llcrypt registered [ 5292.100438] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5293.567619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5293.631637] alg: No test for adler32 (adler32-zlib) [ 5295.185795] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 5295.558368] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5297.329981] Key type lgssc registered [ 5299.011717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5307.253483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5307.279472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5308.721912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.684133] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5319.141719] LustreError: 189004:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5319.164245] LustreError: 189004:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5323.505466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.949709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5329.054377] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5337.286653] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.602113] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5351.946049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.362272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5354.442088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5358.729608] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5367.629116] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5374.948101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5374.957479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.979392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5375.847677] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 5379.044158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.050329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5379.050908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.072841] Lustre: Skipped 1 previous similar message [ 5380.719327] Lustre: server umount lustre-OST0000 complete [ 5389.395507] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b9505741800.ost_server_uuid 50 [ 5391.474188] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5405.671712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5405.679727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.696715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5408.740108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.832835] Lustre: server umount lustre-MDT0000 complete [ 5413.860613] LustreError: 190344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.890679] LustreError: 190344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5415.729146] LustreError: 188984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688483 with bad export cookie 17284170357601395152 [ 5415.734920] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5415.747152] LustreError: 188984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5415.977198] Lustre: server umount lustre-MDT0001 complete [ 5425.035223] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:41:31 (1789688491) [ 5433.209422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.720532] LustreError: 192920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.770701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5438.845795] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5438.952309] LustreError: 192921:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.070151] LustreError: 192920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.947701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.321234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.779255] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5464.582634] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5471.766571] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5480.524046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5480.962807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.165962] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5499.391572] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5504.527484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5508.719398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5509.052606] Lustre: lustre-OST0001: new disk, initializing [ 5509.056610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5509.063237] Lustre: 195761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5509.142851] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5516.428124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5516.461473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5516.716830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5518.504180] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5531.474570] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5537.253840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.259794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5537.262881] Lustre: Skipped 1 previous similar message [ 5537.287300] Lustre: Skipped 3 previous similar messages [ 5542.212346] Lustre: server umount lustre-OST0000 complete [ 5542.391759] LustreError: 194741:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5542.408168] LustreError: 194741:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5552.728559] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5553.162515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5558.744108] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5558.982193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5562.857726] LustreError: 196483:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.888479] LustreError: 196483:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5578.216203] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.216489] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.245043] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5578.258699] LustreError: Skipped 1 previous similar message [ 5578.288964] Lustre: Skipped 2 previous similar messages [ 5579.615313] Lustre: server umount lustre-OST0001 complete [ 5587.425894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5590.520977] Lustre: server umount lustre-MDT0000 complete [ 5595.291294] LustreError: 192900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688662 with bad export cookie 17284170357601396223 [ 5595.294198] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5595.305637] LustreError: 192900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5595.896340] Lustre: server umount lustre-MDT0001 complete [ 5626.672368] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:44:52 (1789688692) [ 5634.591471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.810257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5634.854320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5635.046464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5642.388891] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5653.348239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.473463] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5653.544387] Lustre: 199205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5658.859148] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5668.241893] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5674.938897] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5683.160507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5683.424967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5689.502106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5690.325540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5699.395605] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5706.961337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5707.016903] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5707.217510] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5707.221664] Lustre: Skipped 2 previous similar messages [ 5713.600964] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5721.908609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5725.668041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5725.684166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5725.695586] Lustre: Skipped 3 previous similar messages [ 5725.700753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5725.706449] Lustre: Skipped 3 previous similar messages [ 5731.464607] Lustre: server umount lustre-OST0000 complete [ 5733.168342] LustreError: 201057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.208193] LustreError: 201057:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5749.221109] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.602746] Lustre: server umount lustre-OST0001 complete [ 5759.970346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.988579] Lustre: Skipped 3 previous similar messages [ 5759.998939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.005509] Lustre: Skipped 6 previous similar messages [ 5763.838717] Lustre: server umount lustre-MDT0000 complete [ 5767.597861] LustreError: 202095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789688835 with bad export cookie 17284170357601397630 [ 5767.613257] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5767.889370] Lustre: server umount lustre-MDT0001 complete [ 5793.076923] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:47:39 (1789688859) [ 5812.042878] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5823.256549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5861.839908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.067029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5862.099977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5862.213561] Lustre: lustre-MDT0000: new disk, initializing [ 5862.331433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.377991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5867.219899] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5878.608629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.741945] Lustre: 207303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5878.788337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5878.794588] Lustre: Skipped 1 previous similar message [ 5878.875486] Lustre: lustre-MDT0001: new disk, initializing [ 5878.993821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5879.002727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5883.397150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5894.231619] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5900.562820] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5907.240041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5907.497210] Lustre: lustre-OST0000: new disk, initializing [ 5907.504474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5907.513840] Lustre: 209272:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5913.172351] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5914.700801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5914.710263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5914.815848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5922.108172] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5927.020762] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5927.209339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5931.759194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5931.964697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5935.584966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5935.592268] LustreError: Skipped 1 previous similar message [ 5935.592561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.606032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5935.621535] Lustre: Skipped 2 previous similar messages [ 5935.634550] Lustre: Skipped 1 previous similar message [ 5947.872171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5948.137777] Lustre: server umount lustre-OST0000 complete [ 5958.737169] Lustre: server umount lustre-MDT0000 complete [ 5961.189162] LustreError: 209737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.212246] LustreError: 209737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 5962.482494] LustreError: 209271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689029 with bad export cookie 17284170357601399191 [ 5962.488302] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5962.494878] LustreError: 209271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5962.844865] Lustre: server umount lustre-MDT0001 complete [ 5972.118065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.259834] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5985.792046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5990.970312] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5998.683303] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6004.941138] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6011.997378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.005450] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6028.331846] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.497309] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6033.898797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6039.270352] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6039.453627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6043.109095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6043.120566] LustreError: Skipped 1 previous similar message [ 6048.139431] Lustre: server umount lustre-OST0000 complete [ 6055.436290] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6061.063543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6090.544758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.113997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.127661] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6097.043600] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6104.941229] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6108.923814] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6111.542854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.562392] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6116.834150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.851376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6116.856119] Lustre: Skipped 5 previous similar messages [ 6116.877612] Lustre: Skipped 10 previous similar messages [ 6119.343544] Lustre: server umount lustre-OST0000 complete [ 6129.495771] Lustre: server umount lustre-MDT0000 complete [ 6132.197784] LustreError: 211487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.216647] LustreError: 211487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6133.462042] LustreError: 211475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689200 with bad export cookie 17284170357601400073 [ 6133.474692] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6133.823788] Lustre: server umount lustre-MDT0001 complete [ 6142.552936] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:53:29 (1789689209) [ 6149.085266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.802656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.810559] Lustre: Skipped 6 previous similar messages [ 6154.772790] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6163.747659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.877622] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6176.970079] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6183.168229] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.106354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6196.467545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6204.366564] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6209.149029] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6209.304992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6213.716153] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6213.993738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6220.728853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6220.953592] Lustre: lustre-OST0001: new disk, initializing [ 6220.960033] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6220.967377] Lustre: 220833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6227.669446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6229.112048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6229.117381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6229.232498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6237.110809] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6242.704694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6243.023375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6247.733244] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6248.021799] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6255.088107] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.107182] LustreError: Skipped 5 previous similar messages [ 6264.288707] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6264.460060] Lustre: server umount lustre-OST0001 complete [ 6271.793873] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6272.021771] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6276.856229] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6277.096783] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6307.204427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.934830] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6312.944641] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6312.946222] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6314.297669] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6322.151920] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6323.917620] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6340.065038] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6340.190243] Lustre: server umount lustre-OST0001 complete [ 6363.536672] Lustre: server umount lustre-OST0000 complete [ 6374.501885] Lustre: server umount lustre-MDT0000 complete [ 6378.090176] LustreError: 217443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689445 with bad export cookie 17284170357601401039 [ 6378.094028] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6378.105332] LustreError: 217443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6384.523409] Lustre: server umount lustre-MDT0001 complete [ 6408.420504] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:57:54 (1789689474) [ 6415.916861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.486534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6416.559684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6421.528677] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6431.914815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6432.074767] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6432.119897] Lustre: 226081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6438.033920] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6448.101846] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6455.247770] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6462.629097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.852123] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6469.844318] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6478.584080] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.313247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.504045] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6495.665443] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6504.371765] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6506.219616] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6511.944604] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6512.315977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6517.917114] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6518.124509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6525.170645] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b950667a000.ost_server_uuid 50 [ 6527.893820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b950667a000.ost_server_uuid in IDLE state after 0 sec [ 6529.459947] Lustre: Permanently deactivating lustre-OST0001 [ 6529.498104] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6534.603759] Lustre: Permanently reactivating lustre-OST0001 [ 6534.622354] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6534.628286] Lustre: Skipped 2 previous similar messages [ 6541.294637] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6541.294674] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.301498] LustreError: Skipped 4 previous similar messages [ 6541.309913] Lustre: Skipped 10 previous similar messages [ 6541.317967] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6541.324286] Lustre: Skipped 19 previous similar messages [ 6546.407797] LustreError: 228305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.414210] Lustre: server umount lustre-OST0001 complete [ 6546.432310] LustreError: 228305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [ 6557.695771] Lustre: server umount lustre-OST0000 complete [ 6569.107776] Lustre: server umount lustre-MDT0000 complete [ 6573.229175] LustreError: 226075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689640 with bad export cookie 17284170357601402327 [ 6573.254154] LustreError: 226075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6573.258300] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6573.878186] Lustre: server umount lustre-MDT0001 complete [ 6603.527744] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:01:09 (1789689669) [ 6611.104702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.586588] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6611.609635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6611.616279] Lustre: Skipped 2 previous similar messages [ 6616.310469] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6625.416304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.556460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6625.578947] Lustre: 232796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6630.703131] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6639.683653] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6646.416077] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6653.872753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6654.098887] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6654.136400] Lustre: Permanently deactivating lustre-OST0000 [ 6654.173880] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6660.704861] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6661.694249] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6669.839696] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.395450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6679.478075] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6679.732252] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6679.741131] Lustre: Skipped 11 previous similar messages [ 6686.827808] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6687.491599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6696.086517] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6698.437650] Lustre: Permanently reactivating lustre-OST0000 [ 6698.464413] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6698.474097] Lustre: Skipped 2 previous similar messages [ 6704.300053] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6704.303514] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000ff351ea4, cur 1789689771 deadline 1789689759 last 1789689729 [ 6704.313703] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6704.356029] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6704.376685] Lustre: Skipped 1 previous similar message [ 6704.472371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6721.417782] Lustre: server umount lustre-OST0001 complete [ 6732.589801] Lustre: server umount lustre-OST0000 complete [ 6743.706500] Lustre: server umount lustre-MDT0000 complete [ 6748.111781] LustreError: 232789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789689815 with bad export cookie 17284170357601404007 [ 6748.119272] LustreError: 232789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6748.119296] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6748.522411] Lustre: server umount lustre-MDT0001 complete [ 6759.952631] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6764.509899] Key type lgssc unregistered [ 6764.872135] LNet: 237678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6764.877774] LNetError: 237678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6765.926909] LNet: Removed LNI 192.168.204.107@tcp [ 6767.473525] Key type .llcrypt unregistered [ 6767.478380] Key type ._llcrypt unregistered [ 6796.094434] Key type ._llcrypt registered [ 6796.098229] Key type .llcrypt registered [ 6796.259685] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6798.139730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6798.206575] alg: No test for adler32 (adler32-zlib) [ 6799.353887] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 6799.749850] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6801.603494] Key type lgssc registered [ 6803.324976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6816.001619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6816.026442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.570562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.990464] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6827.008192] LustreError: 239144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6827.043635] LustreError: 239144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6832.106331] LustreError: 239143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.957376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6834.097260] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6834.128721] Lustre: 239132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6834.136278] Lustre: Permanently deactivating lustre-MDT0001 [ 6834.148204] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6834.518417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.879610] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6835.888339] LustreError: 239174:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6835.915386] LustreError: 239174:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6835.932452] LustreError: 239174:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 6839.781620] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6848.402874] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6854.855293] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6862.859657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.491966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.476283] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6878.826866] LustreError: 240985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.850456] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6881.855155] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.325034] LustreError: 240985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.421666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6889.511990] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6889.879292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6895.099722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6895.103415] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6896.965291] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6906.712612] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.901868] Lustre: Permanently reactivating lustre-MDT0001 [ 6909.933793] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6909.943542] Lustre: Skipped 2 previous similar messages [ 6912.175956] Lustre: lustre-MDT0001: Client cc516027-1f39-44df-a2ca-d1a38ea6ecd9 (at 192.168.204.7@tcp) reconnecting [ 6919.380199] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6921.351988] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6927.150147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6927.455312] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6933.356667] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6933.575114] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6936.858437] Lustre: Permanently deactivating lustre-MDT0001 [ 6936.886753] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6936.894978] Lustre: Skipped 2 previous similar messages [ 6945.392977] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6952.278979] Lustre: server umount lustre-MDT0000 complete [ 6952.432493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6952.436371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.468127] LustreError: 239144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6952.484482] LustreError: 239144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6956.524939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.550898] Lustre: Skipped 2 previous similar messages [ 6956.576857] LustreError: 241999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690023 with bad export cookie 9042094958659001654 [ 6956.582241] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.598591] LustreError: 241999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6957.030903] Lustre: server umount lustre-MDT0001 complete [ 6961.374943] Lustre: server umount lustre-OST0000 complete [ 6975.310372] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:07:21 (1789690041) [ 6987.028450] Lustre: server umount lustre-OST0001 complete [ 6994.829380] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7005.876700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7051.271420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7051.770442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7051.791684] Lustre: Skipped 2 previous similar messages [ 7051.829533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7051.983975] Lustre: lustre-MDT0000: new disk, initializing [ 7052.159807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7052.199148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7056.948798] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7070.453139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.603429] Lustre: 247403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7070.718357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7070.730901] Lustre: Skipped 1 previous similar message [ 7070.840788] Lustre: lustre-MDT0001: new disk, initializing [ 7070.992777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.053429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7071.072209] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7076.291114] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7087.515070] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7094.663567] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7103.056803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7103.438695] Lustre: lustre-OST0000: new disk, initializing [ 7103.451511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7103.469064] Lustre: 249375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7103.636104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7111.796124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7111.823775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7111.986496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7112.084726] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7125.036859] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7128.767091] LustreError: 247412:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7138.335723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7138.574318] Lustre: lustre-OST0001: new disk, initializing [ 7138.577673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7138.583164] Lustre: 250489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7138.674458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.849073] LustreError: 247412:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7138.882317] LustreError: 247410:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7146.363957] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7146.706557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7146.726180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7146.891940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7148.988048] LustreError: 247410:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7158.833345] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7160.885704] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7167.474124] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7167.494855] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7167.500081] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7167.515527] Lustre: Skipped 1 previous similar message [ 7169.525635] Lustre: server umount lustre-OST0001 complete [ 7172.594401] LustreError: 249359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.609051] LustreError: 249359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7187.955172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.966085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7187.974201] Lustre: Skipped 1 previous similar message [ 7187.988370] Lustre: Skipped 2 previous similar messages [ 7193.066255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7193.072364] LustreError: 249360:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7193.073782] Lustre: Skipped 1 previous similar message [ 7193.113955] LustreError: 249360:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 7193.485643] Lustre: server umount lustre-OST0000 complete [ 7198.182914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.199699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.211614] Lustre: Skipped 1 previous similar message [ 7203.297897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7203.312801] Lustre: Skipped 1 previous similar message [ 7204.542639] Lustre: server umount lustre-MDT0000 complete [ 7208.502992] LustreError: 247397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690275 with bad export cookie 9042094958659004468 [ 7208.508968] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.511087] LustreError: 247397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7214.853607] Lustre: server umount lustre-MDT0001 complete [ 7226.068584] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7228.918948] Key type lgssc unregistered [ 7229.239576] LNet: 252525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7229.259989] LNetError: 252525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7230.312871] LNet: Removed LNI 192.168.204.107@tcp [ 7231.217902] Key type .llcrypt unregistered [ 7231.219992] Key type ._llcrypt unregistered [ 7273.911268] Key type ._llcrypt registered [ 7273.913765] Key type .llcrypt registered [ 7274.076919] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7275.623537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7275.679297] alg: No test for adler32 (adler32-zlib) [ 7276.932906] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7277.232672] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7278.904226] Key type lgssc registered [ 7280.385830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7289.349264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7289.364353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.602948] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7290.634495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7290.798966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7295.919186] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7304.454464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7304.551923] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7304.582220] Lustre: 254733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7304.899092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7309.034154] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7318.576256] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.602860] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7333.540731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.689095] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7333.930230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.731795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7341.574372] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7351.727955] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.052128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7406.057922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.080506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.570356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7406.571959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.623944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7409.796481] Lustre: server umount lustre-OST0000 complete [ 7414.543820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7428.332221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7428.621343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.431900] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7442.765027] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7446.968391] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7446.987546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7446.995877] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7447.004857] Lustre: Skipped 1 previous similar message [ 7482.852881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7482.855390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.862110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7482.883270] Lustre: Skipped 1 previous similar message [ 7488.192655] Lustre: server umount lustre-OST0000 complete [ 7493.095468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.100722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7493.101853] Lustre: Skipped 1 previous similar message [ 7493.104688] Lustre: Skipped 1 previous similar message [ 7498.214407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7498.245319] Lustre: Skipped 1 previous similar message [ 7499.372398] Lustre: server umount lustre-MDT0000 complete [ 7503.329348] LustreError: 254746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.348830] LustreError: 254746:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7504.938802] LustreError: 254726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690572 with bad export cookie 16988279042447420348 [ 7504.955143] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7505.580555] Lustre: server umount lustre-MDT0001 complete [ 7521.158593] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7524.895520] Key type lgssc unregistered [ 7525.323570] LNet: 260167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7525.340198] LNetError: 260167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7525.359957] LNet: Removed LNI 192.168.204.107@tcp [ 7526.639462] Key type .llcrypt unregistered [ 7526.646290] Key type ._llcrypt unregistered [ 7558.613051] Key type ._llcrypt registered [ 7558.617209] Key type .llcrypt registered [ 7558.766521] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7559.649930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7559.821041] alg: No test for adler32 (adler32-zlib) [ 7560.974553] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7561.277797] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7562.968265] Key type lgssc registered [ 7564.175143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7572.525824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7572.565308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.012446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.040426] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7584.236071] LustreError: 261582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7584.255618] LustreError: 261582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7589.346982] LustreError: 261581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.413449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.784918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7595.216857] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7604.630252] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.232054] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7618.163227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7618.653235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.717774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7624.310826] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7631.797483] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.025322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7661.033480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7661.038967] LustreError: Skipped 1 previous similar message [ 7661.042684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7661.051331] Lustre: Skipped 1 previous similar message [ 7666.168546] Lustre: server umount lustre-OST0000 complete [ 7671.778568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7671.792974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.834440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.846119] Lustre: Skipped 1 previous similar message [ 7676.395428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7676.401458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.906386] Lustre: server umount lustre-MDT0000 complete [ 7681.508303] LustreError: 261582:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.543449] LustreError: 261582:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7683.061753] LustreError: 261563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690750 with bad export cookie 1373304545779720398 [ 7683.072449] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7683.079027] LustreError: 261563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7683.664496] Lustre: server umount lustre-MDT0001 complete [ 7694.707708] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7698.553359] Key type lgssc unregistered [ 7698.932790] LNet: 265890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.945563] LNetError: 265890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7698.964340] LNet: Removed LNI 192.168.204.107@tcp [ 7699.876568] Key type .llcrypt unregistered [ 7699.878725] Key type ._llcrypt unregistered [ 7718.308851] Key type ._llcrypt registered [ 7718.317182] Key type .llcrypt registered [ 7718.475865] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7719.597295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7719.700658] alg: No test for adler32 (adler32-zlib) [ 7720.859593] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7721.146822] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7722.872256] Key type lgssc registered [ 7724.205841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7732.844498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7732.878698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7734.541698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.946473] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7743.972770] LustreError: 267107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.989774] LustreError: 267107:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7749.092399] LustreError: 267108:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7750.790926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7751.173259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7757.148400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7766.773945] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7773.786259] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.574570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.011082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.908697] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7798.186457] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.568917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7817.186211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.196078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.223825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7818.818618] Lustre: server umount lustre-OST0000 complete [ 7825.378091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7825.384226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.406380] Lustre: Skipped 1 previous similar message [ 7825.414314] Lustre: Skipped 1 previous similar message [ 7830.498271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.510109] Lustre: Skipped 1 previous similar message [ 7833.794641] Lustre: server umount lustre-MDT0000 complete [ 7835.622266] LustreError: 267836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7835.647334] LustreError: 267836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7838.773772] LustreError: 267086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789690906 with bad export cookie 10647172878987853425 [ 7838.799733] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7838.804383] LustreError: 267086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7839.424386] Lustre: server umount lustre-MDT0001 complete [ 7854.453860] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7857.885372] Key type lgssc unregistered [ 7858.270646] LNet: 270810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7858.283886] LNetError: 270810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7858.302274] LNet: Removed LNI 192.168.204.107@tcp [ 7859.429478] Key type .llcrypt unregistered [ 7859.434638] Key type ._llcrypt unregistered [ 7887.204642] Key type ._llcrypt registered [ 7887.207095] Key type .llcrypt registered [ 7887.403350] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7889.154924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7889.176444] alg: No test for adler32 (adler32-zlib) [ 7890.309825] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 7890.581857] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7892.320365] Key type lgssc registered [ 7893.748809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7904.779903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7904.806197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.396661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.642596] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7916.514450] LustreError: 272224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.546088] LustreError: 272224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7921.637695] LustreError: 272225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.826846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7924.538116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7931.562825] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7941.304130] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7948.403758] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7956.113647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.449966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.769468] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7971.328860] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7975.924954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8004.577513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.588899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.622058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.625952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8006.640647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.642481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8006.670040] Lustre: Skipped 1 previous similar message [ 8009.925754] Lustre: server umount lustre-OST0000 complete [ 8016.870482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8016.876721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8016.884629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8016.908983] Lustre: Skipped 1 previous similar message [ 8020.385843] Lustre: server umount lustre-MDT0000 complete [ 8021.986300] LustreError: 272952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8022.005428] LustreError: 272952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8024.716462] LustreError: 272204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691092 with bad export cookie 9281494474449126640 [ 8024.754941] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8025.305617] Lustre: server umount lustre-MDT0001 complete [ 8037.822882] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8040.829991] Key type lgssc unregistered [ 8041.138274] LNet: 276532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8041.151093] LNetError: 276532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8041.181311] LNet: Removed LNI 192.168.204.107@tcp [ 8042.076338] Key type .llcrypt unregistered [ 8042.079521] Key type ._llcrypt unregistered [ 8062.436773] Key type ._llcrypt registered [ 8062.438925] Key type .llcrypt registered [ 8062.628132] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8064.212561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8064.263863] alg: No test for adler32 (adler32-zlib) [ 8065.337452] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 8065.616325] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8067.280250] Key type lgssc registered [ 8068.599773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8078.276560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8078.306284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.197718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.968312] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8090.603930] LustreError: 277767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8090.642726] LustreError: 277767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8095.716047] LustreError: 277769:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.356375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.960302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8103.407477] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8112.249129] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8119.137780] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8127.596334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.867552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.523540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8143.774561] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8151.457531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8172.017924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.040879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8172.042923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8175.014718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8175.048329] Lustre: Skipped 2 previous similar messages [ 8177.738901] Lustre: server umount lustre-OST0000 complete [ 8184.801710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8184.818942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.836342] Lustre: Skipped 1 previous similar message [ 8184.844353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.144538] Lustre: server umount lustre-MDT0000 complete [ 8190.434919] LustreError: 277771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.460588] LustreError: 277771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8195.460505] LustreError: 277732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691262 with bad export cookie 14831141037626400672 [ 8195.469970] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8195.486927] LustreError: 277732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8196.265205] Lustre: server umount lustre-MDT0001 complete [ 8212.354657] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8216.969175] Key type lgssc unregistered [ 8217.239672] LNet: 281491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8217.252442] LNetError: 281491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8217.265049] LNet: Removed LNI 192.168.204.107@tcp [ 8218.176283] Key type .llcrypt unregistered [ 8218.179679] Key type ._llcrypt unregistered [ 8232.963406] Key type ._llcrypt registered [ 8232.970913] Key type .llcrypt registered [ 8233.182633] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8257.426275] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8276.873936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8278.470808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8278.497270] alg: No test for adler32 (adler32-zlib) [ 8279.985821] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 8280.326060] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8282.160232] Key type lgssc registered [ 8284.007629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8326.884073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8326.959521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.300950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8328.335095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8328.474669] Lustre: lustre-MDT0000: new disk, initializing [ 8328.583037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.603241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8333.553337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8344.662293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.850245] Lustre: 285056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.947912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8344.952986] Lustre: Skipped 1 previous similar message [ 8345.050956] Lustre: lustre-MDT0001: new disk, initializing [ 8345.156803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8345.195481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8345.209162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8350.062913] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8361.801180] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8369.176588] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8377.497978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.746751] Lustre: lustre-OST0000: new disk, initializing [ 8377.750411] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8377.756703] Lustre: 287024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8377.839606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8386.131621] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8387.160440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8387.178857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8387.323655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8397.736158] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8403.005994] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8403.228628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8408.609950] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8408.959475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8413.159816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8413.161682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.166817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8413.169671] LustreError: Skipped 1 previous similar message [ 8413.203542] Lustre: Skipped 1 previous similar message [ 8417.459376] Lustre: server umount lustre-OST0000 complete [ 8423.400739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8423.412532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.422761] Lustre: Skipped 1 previous similar message [ 8423.430472] Lustre: Skipped 1 previous similar message [ 8428.716683] Lustre: server umount lustre-MDT0000 complete [ 8433.519178] LustreError: 285047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691500 with bad export cookie 5728942699879192828 [ 8433.526722] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8433.535257] LustreError: 285047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8434.104523] Lustre: server umount lustre-MDT0001 complete [ 8448.184412] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:31:54 (1789691514) [ 8458.165078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8458.730916] LustreError: 289439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.755378] LustreError: 289439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8458.819668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.848193] LustreError: 289440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.657957] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8468.965079] LustreError: 289439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.092496] LustreError: 289440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.019858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.500460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8482.113259] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8491.171384] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8499.083339] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8507.602089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8507.998150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.090194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8525.137799] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8545.186748] Lustre: server umount lustre-OST0000 complete [ 8550.382328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.386788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.408580] Lustre: Skipped 1 previous similar message [ 8550.429081] Lustre: Skipped 2 previous similar messages [ 8555.493185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8555.506077] Lustre: Skipped 1 previous similar message [ 8560.608199] Lustre: 283279:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789691612/real 1789691612] req@00000000327b29e9 x1876627393951616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789691628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8560.628821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8560.650666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8560.686150] Lustre: Skipped 1 previous similar message [ 8560.893453] Lustre: server umount lustre-MDT0000 complete [ 8562.659554] Lustre: 283276:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789691614/real 1789691614] req@000000003ac8bbd6 x1876627393952256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789691630 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8565.731502] LustreError: 289439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.758765] LustreError: 289439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8565.805204] Lustre: 283279:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789691617/real 1789691617] req@00000000df137b89 x1876627393953920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789691633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8565.846238] LustreError: 289421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691633 with bad export cookie 5728942699879193710 [ 8565.860625] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8566.228784] Lustre: server umount lustre-MDT0001 complete [ 8579.681455] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8582.720317] Key type lgssc unregistered [ 8583.116824] LNet: 292998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8583.128245] LNetError: 292998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8583.153859] LNet: Removed LNI 192.168.204.107@tcp [ 8584.128316] Key type .llcrypt unregistered [ 8584.134119] Key type ._llcrypt unregistered [ 8614.283446] Key type ._llcrypt registered [ 8614.285127] Key type .llcrypt registered [ 8614.379828] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8633.655083] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8635.102896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8635.283690] alg: No test for adler32 (adler32-zlib) [ 8636.444897] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 8636.647699] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8638.457385] Key type lgssc registered [ 8639.914141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8692.325705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8692.377526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.714699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8693.740470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8693.818713] Lustre: lustre-MDT0000: new disk, initializing [ 8693.908035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.935338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8699.136763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8711.400599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8711.518223] Lustre: 296827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8711.583175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8711.588140] Lustre: Skipped 1 previous similar message [ 8711.725125] Lustre: lustre-MDT0001: new disk, initializing [ 8711.889897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8711.941237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8711.957076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8716.774638] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8727.268608] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8734.768875] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8742.509253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.738577] Lustre: lustre-OST2710: new disk, initializing [ 8742.741912] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8742.747571] Lustre: 298793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8742.843237] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8748.130842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8748.148466] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8748.306971] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8749.969499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8759.921556] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8767.844028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.092941] Lustre: lustre-OST03e8: new disk, initializing [ 8768.097362] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8768.109711] Lustre: 299860:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.259390] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8775.300434] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8777.887945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8777.898822] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8777.967406] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8783.705512] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8792.142086] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8792.451429] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8797.337206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8797.624371] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8802.524794] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8802.761727] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8807.348505] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8807.553810] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8814.066541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8814.199902] Lustre: server umount lustre-MDT0000 complete [ 8818.219394] LustreError: 299859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789691885 with bad export cookie 15761472008288562201 [ 8818.223653] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8818.229155] LustreError: 299859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8818.528656] Lustre: server umount lustre-MDT0001 complete [ 8831.991835] Lustre: server umount lustre-OST2710 complete [ 8835.360463] Lustre: 294621:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789691886/real 1789691886] req@000000006f9aaedd x1876627768281728/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789691902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8835.385555] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.395502] Lustre: Skipped 3 previous similar messages [ 8835.745494] Lustre: server umount lustre-OST03e8 complete [ 8855.440471] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8866.532294] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8904.401151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8904.704522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8904.771815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8904.882745] Lustre: lustre-MDT0000: new disk, initializing [ 8904.966371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.996988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8909.285711] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8919.882276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.071345] Lustre: 305105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8920.115386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8920.120225] Lustre: Skipped 1 previous similar message [ 8920.247666] Lustre: lustre-MDT0001: new disk, initializing [ 8920.391939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8920.457749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8920.478217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8924.907743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8933.579767] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8939.879450] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8947.511124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8947.788376] Lustre: lustre-OST0000: new disk, initializing [ 8947.795565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8947.802525] Lustre: 307078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8947.898460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8953.418384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8953.433321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8953.593793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8954.683042] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8963.894415] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8968.478201] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8968.658921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8972.218774] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8972.396848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8974.308843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.313137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8974.325988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8974.329676] Lustre: Skipped 1 previous similar message [ 8974.365766] LustreError: Skipped 1 previous similar message [ 8979.428580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8979.434791] Lustre: Skipped 2 previous similar messages [ 8984.549566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8984.562229] Lustre: Skipped 1 previous similar message [ 8988.128582] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8988.302821] Lustre: server umount lustre-OST0000 complete [ 8992.225154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8992.229603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.236479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8998.051322] Lustre: server umount lustre-MDT0000 complete [ 8999.909690] LustreError: 306434:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.926378] LustreError: 306434:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9001.266418] LustreError: 307077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692068 with bad export cookie 15761472008288563881 [ 9001.269880] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.282447] LustreError: 307077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9001.615776] Lustre: server umount lustre-MDT0001 complete [ 9009.146165] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:41:15 (1789692075) [ 9010.602943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9012.121728] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:41:18 (1789692078) [ 9015.246639] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9039.163327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.420578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9039.445729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9039.581765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9043.623744] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9051.866348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.930304] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9051.954430] Lustre: 310936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9055.908642] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9076.014304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9076.204481] Lustre: Denying initial registration attempt from nid 192.168.204.107@tcp, specified as failover [ 9076.214402] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9076.224570] LustreError: 312387:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9076.230356] LustreError: 312387:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9076.235173] LustreError: 312387:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9076.294829] Lustre: server umount lustre-OST0000 complete [ 9076.298299] LustreError: 312387:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9082.850346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9082.855790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.869164] Lustre: Skipped 1 previous similar message [ 9082.875785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9082.880722] Lustre: Skipped 2 previous similar messages [ 9087.602692] Lustre: server umount lustre-MDT0000 complete [ 9089.508291] LustreError: 310947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9089.519299] LustreError: 310947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9090.499020] LustreError: 310927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692157 with bad export cookie 15761472008288564763 [ 9090.507738] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9090.515088] LustreError: 310927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9090.744408] Lustre: server umount lustre-MDT0001 complete [ 9099.008256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9101.151250] Key type lgssc unregistered [ 9101.394714] LNet: 313508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9101.399197] LNetError: 313508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9101.411130] LNet: Removed LNI 192.168.204.107@tcp [ 9102.056218] Key type .llcrypt unregistered [ 9102.058934] Key type ._llcrypt unregistered [ 9111.376967] Key type ._llcrypt registered [ 9111.380162] Key type .llcrypt registered [ 9111.479789] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9112.270958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9112.292515] alg: No test for adler32 (adler32-zlib) [ 9113.313550] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 9113.518448] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9115.196811] Key type lgssc registered [ 9116.102534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9157.362428] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9164.532209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9164.550579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.769399] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9165.806246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9165.936873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.837551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9177.938109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.042067] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9178.073196] Lustre: 316264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.329049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9182.429542] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9189.374590] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9195.235794] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9214.186176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9214.335163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9214.534863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9219.518931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9225.684729] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9230.821687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.822815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9230.842740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9230.853637] Lustre: Skipped 1 previous similar message [ 9233.675425] Lustre: server umount lustre-OST0000 complete [ 9240.034710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9240.039776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9240.050912] Lustre: Skipped 1 previous similar message [ 9240.054624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.072281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.073171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9241.102043] Lustre: Skipped 1 previous similar message [ 9242.923644] Lustre: server umount lustre-MDT0000 complete [ 9250.464566] LustreError: 316257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692317 with bad export cookie 7340915546476574037 [ 9250.468919] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.476485] LustreError: 316257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9250.852878] Lustre: server umount lustre-MDT0001 complete [ 9274.327942] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9278.972660] Key type lgssc unregistered [ 9279.436804] LNet: 319863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9279.451313] LNetError: 319863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9279.476864] LNet: Removed LNI 192.168.204.107@tcp [ 9280.673680] Key type .llcrypt unregistered [ 9280.677690] Key type ._llcrypt unregistered [ 9312.050617] Key type ._llcrypt registered [ 9312.052497] Key type .llcrypt registered [ 9312.191739] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9313.043388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9313.058577] alg: No test for adler32 (adler32-zlib) [ 9314.215382] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 9314.633951] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9316.266526] Key type lgssc registered [ 9317.687474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9327.192351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9327.244321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.929866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9334.212661] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9339.365670] LustreError: 321279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9339.396029] LustreError: 321279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9343.138422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.472237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.853610] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9356.601977] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9363.273520] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9372.600050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.996779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.189503] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9388.565600] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9398.757406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.764125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9398.785593] Lustre: Skipped 2 previous similar messages [ 9398.808734] Lustre: Skipped 2 previous similar messages [ 9400.556047] Lustre: server umount lustre-MDT0000 complete [ 9400.660420] LustreError: 321274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.880748] LustreError: 321276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9403.902309] LustreError: 321276:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9405.314065] LustreError: 321260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692472 with bad export cookie 8047684352385305160 [ 9405.332866] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9405.341020] LustreError: 321260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9405.771773] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 9408.995185] LustreError: 323632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9408.996761] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.017770] LustreError: 323632:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9409.091185] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9411.781853] Lustre: server umount lustre-MDT0001 complete [ 9420.099094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9430.983287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.956192] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6faf1e47b065664f [ 9449.974911] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 9450.467711] LustreError: 323110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9450.492469] LustreError: 323110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9450.640846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.738950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9455.182526] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9461.095305] LustreError: 324900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.121720] LustreError: 324900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9464.787951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.083328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9466.109555] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9469.277393] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9478.335835] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9480.019385] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9485.371085] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9487.141587] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9496.036650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9496.046343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9496.056403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9496.071571] Lustre: Skipped 2 previous similar messages [ 9496.545307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9498.639366] Lustre: server umount lustre-OST0000 complete [ 9506.273670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9506.281148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9506.301490] Lustre: Skipped 1 previous similar message [ 9506.314136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9506.327225] Lustre: Skipped 3 previous similar messages [ 9509.085525] Lustre: server umount lustre-MDT0000 complete [ 9512.417546] LustreError: 326766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.436699] LustreError: 326766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9513.376246] LustreError: 321262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692580 with bad export cookie 8047684352385312335 [ 9513.388489] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9513.747616] Lustre: server umount lustre-MDT0001 complete [ 9525.774420] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9528.730800] Key type lgssc unregistered [ 9529.117194] LNet: 327547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9529.137734] LNetError: 327547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9530.214800] LNet: Removed LNI 192.168.204.107@tcp [ 9531.470271] Key type .llcrypt unregistered [ 9531.476066] Key type ._llcrypt unregistered [ 9558.698471] Key type ._llcrypt registered [ 9558.700456] Key type .llcrypt registered [ 9558.864392] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9559.860699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9559.960518] alg: No test for adler32 (adler32-zlib) [ 9561.056513] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 9561.332487] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9563.008235] Key type lgssc registered [ 9564.071740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9572.157126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9572.177827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.635736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9578.496955] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9583.077628] LustreError: 328981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.096165] LustreError: 328981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9587.814788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.203415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9592.857931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9601.785278] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9608.124525] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9613.220779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9613.224365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9613.237492] Lustre: Skipped 1 previous similar message [ 9617.497968] Lustre: server umount lustre-MDT0000 complete [ 9618.404745] LustreError: 329983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9618.443399] LustreError: 329983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9621.986451] LustreError: 328961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692689 with bad export cookie 17633968496573108098 [ 9621.993567] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9621.997809] LustreError: 328961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9622.413101] Lustre: server umount lustre-MDT0001 complete [ 9638.797233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9639.118921] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9639.161393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9639.366734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9644.464304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9654.370402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.427274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9654.447641] Lustre: 331571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9654.656625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9659.890923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9669.351317] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9676.557250] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9687.288415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9687.502798] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9687.519978] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9687.529954] LustreError: 333455:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9687.536381] LustreError: 333455:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9687.541108] LustreError: 333455:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9687.617676] Lustre: server umount lustre-OST0000 complete [ 9687.622668] LustreError: 333455:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9698.041234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9698.417250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9698.650279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9705.343709] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9708.202269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9713.431404] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9722.633346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9722.807892] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9722.977575] Lustre: lustre-OST0001: new disk, initializing [ 9722.983347] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9722.988716] Lustre: 334998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9723.084698] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9729.634399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9729.641115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9729.732077] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9730.030619] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9739.502818] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9745.384872] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.385590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9745.395801] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9745.395812] Lustre: Skipped 1 previous similar message [ 9745.415567] Lustre: Skipped 1 previous similar message [ 9745.450698] LustreError: Skipped 1 previous similar message [ 9750.509536] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9750.519133] Lustre: Skipped 2 previous similar messages [ 9755.627956] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9755.636728] Lustre: Skipped 1 previous similar message [ 9757.152144] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9757.318674] Lustre: server umount lustre-OST0001 complete [ 9760.769240] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9760.783403] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9764.321722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9764.336184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.354818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9765.859209] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.888378] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9770.982630] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9770.998104] LustreError: 335321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9776.096120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9776.102868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9776.118167] Lustre: Skipped 6 previous similar messages [ 9776.341759] Lustre: server umount lustre-OST0000 complete [ 9782.754914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9782.772127] LustreError: Skipped 1 previous similar message [ 9782.776356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9782.788535] Lustre: Skipped 1 previous similar message [ 9789.926961] Lustre: server umount lustre-MDT0000 complete [ 9791.465936] LustreError: 331583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.481949] LustreError: 331583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9794.653899] LustreError: 331562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789692862 with bad export cookie 17633968496573108658 [ 9794.660409] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9794.677203] LustreError: 331562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9795.128547] Lustre: server umount lustre-MDT0001 complete [ 9808.803335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9812.667656] Key type lgssc unregistered [ 9813.230825] LNet: 336960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9813.252757] LNetError: 336960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9813.305490] LNet: Removed LNI 192.168.204.107@tcp [ 9815.016619] Key type .llcrypt unregistered [ 9815.019413] Key type ._llcrypt unregistered [ 9889.671443] Key type ._llcrypt registered [ 9889.673624] Key type .llcrypt registered [ 9889.778950] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9909.558912] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9910.648145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9910.859852] alg: No test for adler32 (adler32-zlib) [ 9912.488852] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [ 9912.908583] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9914.632226] Key type lgssc registered [ 9916.229598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9959.894171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9959.936483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.337852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9961.411638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9961.591980] Lustre: lustre-MDT0000: new disk, initializing [ 9961.694435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.718487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9968.364344] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9979.875056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9980.047097] Lustre: 341894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9980.184738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9980.194510] Lustre: Skipped 1 previous similar message [ 9980.375554] Lustre: lustre-MDT0001: new disk, initializing [ 9980.554272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9980.619371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9980.639052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9986.362461] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9997.815443] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10004.321913] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10012.439696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10012.778981] Lustre: lustre-OST0000: new disk, initializing [10012.783758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10012.795936] Lustre: 343861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10012.961577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10019.947758] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10021.515262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10021.526262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10021.640362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10030.145131] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10035.217290] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10035.590944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10040.947663] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10041.156911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10047.458457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10047.478255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.508102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.513488] Lustre: Skipped 1 previous similar message [10049.587183] Lustre: server umount lustre-OST0000 complete [10057.189781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10057.194525] LustreError: Skipped 1 previous similar message [10057.198671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.211207] Lustre: Skipped 1 previous similar message [10057.215069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10059.609324] Lustre: server umount lustre-MDT0000 complete [10063.785654] LustreError: 342690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693131 with bad export cookie 10619959840943584541 [10063.786565] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.808561] LustreError: 342690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10064.392757] Lustre: server umount lustre-MDT0001 complete [10073.832792] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:59:00 (1789693140) [10081.489954] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:59:07 (1789693147) [10090.615413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.140286] LustreError: 346619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10091.166251] LustreError: 346619:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10091.236953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10096.063494] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10096.613736] LustreError: 346620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.717023] LustreError: 346619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10104.495555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.794297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10109.056655] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10116.604789] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10123.478879] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10130.545401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10130.893250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10137.907460] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10144.931914] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10155.650963] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10157.445725] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10158.910355] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10160.489852] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10162.216919] Lustre: Failing over lustre-MDT0000 [10162.438926] Lustre: server umount lustre-MDT0000 complete [10165.740617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.757690] LustreError: 346615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.763470] Lustre: Skipped 2 previous similar messages [10170.851672] LustreError: 346620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.876342] LustreError: 346620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10180.997258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10181.097849] LustreError: 346620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10181.128937] LustreError: 346620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10181.233496] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.577435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.669034] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10182.454171] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10186.318556] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10186.738820] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10186.784180] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10186.830871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10197.708488] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10199.506856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10203.555862] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10205.834371] Lustre: Failing over lustre-MDT0000 [10206.097877] Lustre: server umount lustre-MDT0000 complete [10207.200847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10207.203626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10207.212528] LustreError: 346616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.261596] Lustre: Skipped 3 previous similar messages [10223.589179] Lustre: 340118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789693274/real 1789693274] req@00000000f34b091e x1876629106286592/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789693290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10223.606682] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10224.494111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.824690] LustreError: 340114:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bbe4fd69 x1876629106291840/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10234.108613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10234.158861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10234.653259] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10238.477314] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10239.470967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10239.476316] Lustre: Skipped 2 previous similar messages [10239.501768] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10239.561547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10246.936477] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10248.207363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10254.822385] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.834456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10254.838288] Lustre: Skipped 2 previous similar messages [10254.840745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10254.847045] Lustre: Skipped 3 previous similar messages [10259.796657] Lustre: server umount lustre-MDT0000 complete [10259.938509] LustreError: 347659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10259.951841] LustreError: 347659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 28 previous similar messages [10263.025420] LustreError: 346599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693330 with bad export cookie 10619959840943587250 [10263.029535] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10263.032580] LustreError: 346599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10263.355370] Lustre: server umount lustre-MDT0001 complete [10277.318828] Lustre: server umount lustre-OST0000 complete [10292.196302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10297.361147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10305.033931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10305.424179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10305.435855] Lustre: Skipped 1 previous similar message [10309.417990] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10318.274620] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10324.096744] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10331.292888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10337.415364] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10337.855531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10344.760727] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10355.182199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10355.185395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10355.186959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10355.212636] Lustre: Skipped 1 previous similar message [10360.289521] LustreError: 355552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.299260] LustreError: 355552:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10360.352737] Lustre: server umount lustre-OST0000 complete [10365.415766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.421784] Lustre: Skipped 1 previous similar message [10375.651363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.655576] Lustre: Skipped 4 previous similar messages [10378.208546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10378.342993] Lustre: server umount lustre-MDT0000 complete [10381.513901] LustreError: 352952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693448 with bad export cookie 10619959840943587999 [10381.519117] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10381.527652] LustreError: 352952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10381.757283] Lustre: server umount lustre-MDT0001 complete [10390.849807] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10393.518308] Key type lgssc unregistered [10393.766925] LNet: 356506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10393.777310] LNetError: 356506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10393.788676] LNet: Removed LNI 192.168.204.107@tcp [10394.443790] Key type .llcrypt unregistered [10394.446479] Key type ._llcrypt unregistered [10419.875179] Key type ._llcrypt registered [10419.877850] Key type .llcrypt registered [10420.002977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10421.124295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10421.137299] alg: No test for adler32 (adler32-zlib) [10422.253221] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [10422.511658] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10424.256400] Key type lgssc registered [10425.401766] Lustre: Echo OBD driver; http://www.lustre.org/ [10433.493284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10433.523907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10435.030446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10439.498644] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10445.306988] LustreError: 358019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10445.330952] LustreError: 358019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10449.437932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.834317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10454.602751] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10461.633452] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10466.984442] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10474.414827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10474.867799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10479.996071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10481.393931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10488.578316] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10500.582306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.583633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10500.592382] Lustre: Skipped 2 previous similar messages [10500.597092] Lustre: Skipped 2 previous similar messages [10503.378531] Lustre: server umount lustre-MDT0000 complete [10505.703139] LustreError: 360124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.711867] LustreError: 360124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10506.221106] LustreError: 358001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693573 with bad export cookie 5111814204921482727 [10506.231666] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10506.493132] Lustre: server umount lustre-MDT0001 complete [10519.691589] Lustre: server umount lustre-OST0000 complete [10531.588166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.077574] LustreError: 361731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.166522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.568452] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10537.450253] LustreError: 361732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10542.563413] LustreError: 361731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10544.626323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.983693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10548.476719] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10554.674763] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10559.805462] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10566.095132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10566.422379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10571.100511] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10576.708701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10577.121830] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10587.108782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10587.113685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10587.122441] Lustre: Skipped 2 previous similar messages [10587.136376] Lustre: Skipped 2 previous similar messages [10592.229374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10592.235181] Lustre: Skipped 2 previous similar messages [10592.857448] Lustre: server umount lustre-MDT0000 complete [10596.162923] LustreError: 361712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693663 with bad export cookie 5111814204921483833 [10596.164759] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10596.174527] LustreError: 361712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10596.466756] Lustre: server umount lustre-MDT0001 complete [10608.906354] Lustre: server umount lustre-OST0000 complete [10622.557216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.964099] LustreError: 365641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10623.014936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10626.982249] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10634.773524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10638.993860] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10645.627289] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10650.922324] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10657.845569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.296867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10658.301708] Lustre: Skipped 1 previous similar message [10664.179837] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10668.537528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10671.026752] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10675.689670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10675.698195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.713944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10678.764572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.766174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10681.058780] Lustre: server umount lustre-OST0000 complete [10686.434818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10686.443938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.458828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.464237] Lustre: Skipped 2 previous similar messages [10690.724796] Lustre: server umount lustre-MDT0000 complete [10694.114572] LustreError: 365636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.138287] LustreError: 365636:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10694.354939] LustreError: 365621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789693761 with bad export cookie 5111814204921484862 [10694.358871] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10694.368174] LustreError: 365621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10694.779736] Lustre: server umount lustre-MDT0001 complete [10703.647591] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10706.134506] Key type lgssc unregistered [10706.369848] LNet: 369169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10706.380147] LNetError: 369169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10706.389962] LNet: Removed LNI 192.168.204.107@tcp [10707.102735] Key type .llcrypt unregistered [10707.105620] Key type ._llcrypt unregistered [10728.456575] Key type ._llcrypt registered [10728.459920] Key type .llcrypt registered [10728.591934] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10729.697149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10729.715590] alg: No test for adler32 (adler32-zlib) [10730.729734] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [10730.943684] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10732.616438] Key type lgssc registered [10733.591692] Lustre: Echo OBD driver; http://www.lustre.org/ [10740.731337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10740.760244] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10740.767396] LustreError: 370599:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10740.792260] LustreError: 370599:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10740.806290] LustreError: 370599:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10740.811218] LustreError: 370599:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10740.818861] LustreError: 370599:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10748.047963] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10748.060425] LustreError: 370996:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10748.082139] LustreError: 370996:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10748.089124] LustreError: 370996:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10748.095325] LustreError: 370996:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10748.103479] LustreError: 370996:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10767.917498] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10770.909867] Key type lgssc unregistered [10771.275559] LNet: 371999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10771.284710] LNetError: 371999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10771.306313] LNet: Removed LNI 192.168.204.107@tcp [10772.343990] Key type .llcrypt unregistered [10772.345763] Key type ._llcrypt unregistered [10797.449782] Key type ._llcrypt registered [10797.454754] Key type .llcrypt registered [10797.637733] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [10817.483578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10819.384908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10819.418279] alg: No test for adler32 (adler32-zlib) [10820.637926] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [10821.111766] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10822.880214] Key type lgssc registered [10824.431652] Lustre: Echo OBD driver; http://www.lustre.org/ [10861.725848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10861.752389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.203825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10863.287679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10863.381102] Lustre: lustre-MDT0000: new disk, initializing [10863.506620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10863.534770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10868.743214] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10881.398122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10881.541526] Lustre: 375158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10881.588956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10881.595813] Lustre: Skipped 1 previous similar message [10881.700653] Lustre: lustre-MDT0001: new disk, initializing [10881.819555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10881.849980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10881.867139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10887.521882] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10899.778687] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10906.695328] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10914.980158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.338885] Lustre: lustre-OST0000: new disk, initializing [10915.349512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10915.361231] Lustre: 377127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10915.488394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10921.638853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10921.648444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10921.964794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10923.808745] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10935.870660] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10941.951544] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10942.387628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10947.658524] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10947.986687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10952.680256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10952.680960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10952.692387] LustreError: Skipped 1 previous similar message [10952.696983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10952.705336] Lustre: Skipped 1 previous similar message [10952.724178] Lustre: Skipped 1 previous similar message [10956.669721] Lustre: server umount lustre-OST0000 complete [10962.914665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10962.943698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10962.958495] Lustre: Skipped 1 previous similar message [10968.001632] Lustre: server umount lustre-MDT0000 complete [10971.905828] LustreError: 375150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694039 with bad export cookie 326875487227802537 [10971.913040] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10971.927057] LustreError: 375150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10972.356766] Lustre: server umount lustre-MDT0001 complete [10982.436728] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10984.512197] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:14:10 (1789694050) [10992.253181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10992.704926] LustreError: 379585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10992.726750] LustreError: 379585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10992.782692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10998.064824] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10998.246276] LustreError: 379586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11003.364974] LustreError: 379585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11007.171260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11007.635952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11011.640070] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11018.745083] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11024.374735] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11032.429728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.728482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11040.172912] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11049.647383] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11058.951922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11059.178555] Lustre: lustre-OST0001: new disk, initializing [11059.185093] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11059.195918] Lustre: 382422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11059.330568] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11067.643824] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11068.647688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11068.664933] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11068.800774] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11080.335442] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.174609] Lustre: lustre-OST0001: Not available for connect from 192.168.204.7@tcp (stopping) [11089.384709] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11089.389989] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11089.394883] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11089.412114] Lustre: Skipped 2 previous similar messages [11090.550795] Lustre: server umount lustre-OST0001 complete [11092.260554] LustreError: 381407:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.634500] LustreError: 381405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.652711] LustreError: 381405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11108.835305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11108.840582] LustreError: Skipped 1 previous similar message [11108.847801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11108.880724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11108.892458] Lustre: Skipped 1 previous similar message [11109.865451] LustreError: 381405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.895793] LustreError: 381405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11114.814852] Lustre: server umount lustre-OST0000 complete [11119.394590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.405349] Lustre: Skipped 2 previous similar messages [11119.409043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11119.417752] Lustre: Skipped 2 previous similar messages [11125.784276] Lustre: server umount lustre-MDT0000 complete [11129.825367] LustreError: 379586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.873598] LustreError: 379586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11130.504782] LustreError: 379566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694197 with bad export cookie 326875487227803419 [11130.515436] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11130.521666] LustreError: 379566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11137.126859] Lustre: server umount lustre-MDT0001 complete [11149.509540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11152.974780] Key type lgssc unregistered [11153.322286] LNet: 384413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11153.345304] LNetError: 384413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11153.370156] LNet: Removed LNI 192.168.204.107@tcp [11154.399567] Key type .llcrypt unregistered [11154.402426] Key type ._llcrypt unregistered [11207.370335] Key type ._llcrypt registered [11207.377096] Key type .llcrypt registered [11207.528772] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11208.779058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11208.790748] alg: No test for adler32 (adler32-zlib) [11209.927646] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11210.182948] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11211.848248] Key type lgssc registered [11213.077945] Lustre: Echo OBD driver; http://www.lustre.org/ [11222.765907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11222.780970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.064645] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11224.110078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11224.393337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11229.237765] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11239.258590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11239.387235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11239.413139] Lustre: 387209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11239.768808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11244.539489] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11253.496789] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11259.832337] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11263.458167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11263.476676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11268.572424] Lustre: server umount lustre-MDT0000 complete [11273.175922] LustreError: 387201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694340 with bad export cookie 6621011282914787516 [11273.180661] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.194475] LustreError: 387201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11273.875736] Lustre: server umount lustre-MDT0001 complete [11285.442270] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:19:11 (1789694351) [11293.262101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.775262] LustreError: 389724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.808983] LustreError: 389724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11293.883027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11298.427193] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11299.297168] LustreError: 389725:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.427932] LustreError: 389724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.070862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11308.413897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11312.931910] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11321.745586] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11327.661526] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11334.354926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11334.604508] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11334.828856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11341.378684] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11348.532820] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11356.386416] Lustre: Permanently deactivating lustre-OST0000 [11356.396763] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11362.884950] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11365.593210] LustreError: 392710:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11365.603671] LustreError: 392710:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11365.614198] LustreError: 392710:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11368.064854] LustreError: 392759:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11368.069731] LustreError: 392759:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11376.355548] Lustre: server umount lustre-OST0000 complete [11379.326941] LustreError: 393008:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11379.331579] LustreError: 393008:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11382.760526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11382.764673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11382.783878] Lustre: Skipped 1 previous similar message [11382.784777] Lustre: Skipped 2 previous similar messages [11387.875172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.880101] Lustre: Skipped 2 previous similar messages [11391.029588] Lustre: server umount lustre-MDT0000 complete [11393.000635] LustreError: 389719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.023265] LustreError: 389719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11394.896603] LustreError: 389704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694462 with bad export cookie 6621011282914788097 [11394.899541] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11394.907742] LustreError: 389704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11405.440549] Lustre: server umount lustre-MDT0001 complete [11412.382547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.195970] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11418.515901] LustreError: 394167:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11424.345718] Lustre: 394505:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.107@tcp [11426.329406] Lustre: server umount MGS complete [11438.299202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.841789] LustreError: 395230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11438.865502] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11438.880193] Lustre: Skipped 1 previous similar message [11438.947257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11442.935256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11449.314383] LustreError: 395230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11449.335336] LustreError: 395230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11450.906844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11451.238235] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11455.647527] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11462.385064] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11467.788351] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11469.389953] Lustre: Permanently reactivating lustre-OST0000 [11469.424994] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11469.434417] Lustre: Skipped 2 previous similar messages [11496.418716] LustreError: 397009:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11504.252032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11504.587672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11504.595732] Lustre: Skipped 1 previous similar message [11509.967491] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11516.425366] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11520.993419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11520.997788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11521.025661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11521.031716] Lustre: Skipped 1 previous similar message [11526.115229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11526.124168] Lustre: Skipped 1 previous similar message [11526.530371] Lustre: server umount lustre-OST0000 complete [11531.235285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.253252] Lustre: Skipped 2 previous similar messages [11535.881711] Lustre: server umount lustre-MDT0000 complete [11536.353090] LustreError: 395231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11536.374557] LustreError: 395231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11538.503668] LustreError: 398690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694605 with bad export cookie 6621011282914789147 [11538.507981] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11538.516234] LustreError: 398690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11538.731931] Lustre: server umount lustre-MDT0001 complete [11547.070215] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11549.558384] Key type lgssc unregistered [11549.795902] LNet: 399642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11549.804756] LNetError: 399642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11549.826658] LNet: Removed LNI 192.168.204.107@tcp [11550.578273] Key type .llcrypt unregistered [11550.580850] Key type ._llcrypt unregistered [11564.056360] Key type ._llcrypt registered [11564.058291] Key type .llcrypt registered [11564.137682] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11576.591464] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11577.272339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11577.290270] alg: No test for adler32 (adler32-zlib) [11578.351649] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11578.553477] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11580.201623] Key type lgssc registered [11581.140547] Lustre: Echo OBD driver; http://www.lustre.org/ [11606.250247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11606.264453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11607.396354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11607.420916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11607.485503] Lustre: lustre-MDT0000: new disk, initializing [11607.526143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11607.539042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11610.849802] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11618.703529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.773606] Lustre: 402774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11618.821272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11618.824591] Lustre: Skipped 1 previous similar message [11618.884137] Lustre: lustre-MDT0001: new disk, initializing [11618.944757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11618.970376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11618.978500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11621.825493] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11628.631749] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11632.655363] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11637.682842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.890314] Lustre: lustre-OST0000: new disk, initializing [11637.895693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11637.901460] Lustre: 404740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11637.970900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11643.018295] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11645.990874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11645.999206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11646.112834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11649.654249] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11653.129411] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11653.352362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11656.772980] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11656.999905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11661.288062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11661.296692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11661.303387] Lustre: Skipped 1 previous similar message [11661.312436] Lustre: Skipped 1 previous similar message [11664.731631] Lustre: server umount lustre-OST0000 complete [11670.501200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11670.505378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11670.513041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.522396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11671.525071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.679344] Lustre: server umount lustre-MDT0000 complete [11675.993387] LustreError: 402809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694743 with bad export cookie 14959032085718950650 [11675.994505] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.000098] LustreError: 402809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11676.194099] Lustre: server umount lustre-MDT0001 complete [11681.437098] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11682.442526] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:25:49 (1789694749) [11686.459675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.818981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11689.297210] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11692.006608] LustreError: 407199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.026338] LustreError: 407199:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11694.869139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11695.118206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11698.575848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11704.027353] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11708.279739] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11712.763477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.034182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11717.102785] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11722.362980] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11723.900697] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11735.475185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11740.641249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11740.648930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.657655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11740.661113] Lustre: Skipped 1 previous similar message [11742.692776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11744.857605] Lustre: server umount lustre-OST0000 complete [11747.812922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.813984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11747.824804] Lustre: Skipped 2 previous similar messages [11747.829134] Lustre: Skipped 2 previous similar messages [11753.726522] Lustre: server umount lustre-MDT0000 complete [11756.777767] LustreError: 407179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694824 with bad export cookie 14959032085718951525 [11756.787407] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11756.798208] LustreError: 407179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11757.196759] Lustre: server umount lustre-MDT0001 complete [11764.480171] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11766.532925] Key type lgssc unregistered [11766.758501] LNet: 410940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11766.766920] LNetError: 410940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11766.778859] LNet: Removed LNI 192.168.204.107@tcp [11767.400812] Key type .llcrypt unregistered [11767.404253] Key type ._llcrypt unregistered [11781.993278] Key type ._llcrypt registered [11781.995808] Key type .llcrypt registered [11782.064298] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11782.294369] Key type .llcrypt unregistered [11782.299919] Key type ._llcrypt unregistered [11792.484476] Key type ._llcrypt registered [11792.486439] Key type .llcrypt registered [11792.545600] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11792.932105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11792.942087] alg: No test for adler32 (adler32-zlib) [11793.836299] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11793.954796] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11795.552214] Key type lgssc registered [11796.192416] Lustre: Echo OBD driver; http://www.lustre.org/ [11800.545789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11800.558161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.837092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11804.294140] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11808.711040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11808.876154] LustreError: 413124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.886815] LustreError: 413124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11808.894095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11811.791074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11811.942460] LustreError: 413123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.949428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11811.950689] LustreError: 413123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11815.321846] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.444767] LustreError: 413124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11817.458666] LustreError: 413124:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11819.027889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11819.260822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.366265] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11832.803434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11832.809071] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.823324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11834.849769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11834.855761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11834.858532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.233642] Lustre: server umount lustre-OST0000 complete [11839.970751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11839.971798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11839.977757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11839.983680] Lustre: Skipped 2 previous similar messages [11844.564139] Lustre: server umount lustre-MDT0000 complete [11845.089903] LustreError: 413123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11845.097467] LustreError: 413123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11846.507513] LustreError: 413105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789694913 with bad export cookie 15730667446019121232 [11846.510236] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11846.514978] LustreError: 413105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11846.717509] Lustre: server umount lustre-MDT0001 complete [11852.510259] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11854.096137] Key type lgssc unregistered [11854.303785] LNet: 416266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11854.308236] LNetError: 416266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11854.325763] LNet: Removed LNI 192.168.204.107@tcp [11854.802684] Key type .llcrypt unregistered [11854.804661] Key type ._llcrypt unregistered [11868.686237] Key type ._llcrypt registered [11868.689777] Key type .llcrypt registered [11868.765186] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11869.157365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11869.169770] alg: No test for adler32 (adler32-zlib) [11870.073950] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [11870.211658] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11871.832524] Key type lgssc registered [11872.527095] Lustre: Echo OBD driver; http://www.lustre.org/ [11877.403735] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11877.414974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11892.896797] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11898.016400] LustreError: 417631:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [11924.640742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11927.898546] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11932.149138] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11935.781113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.036035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11936.072122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11938.189995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11943.038347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.243136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11945.340497] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11949.230884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11952.135214] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11956.707265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11956.714766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11956.725292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11958.756291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.758182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11958.763290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11958.778730] Lustre: Skipped 1 previous similar message [11961.733549] Lustre: server umount lustre-OST0000 complete [11963.873262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11963.878208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.880662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11963.891692] Lustre: Skipped 1 previous similar message [11968.993608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11968.996272] Lustre: Skipped 2 previous similar messages [11974.114314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11974.122442] Lustre: Skipped 1 previous similar message [11978.208388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11978.300198] Lustre: server umount lustre-MDT0000 complete [11979.234350] LustreError: 419911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.243901] LustreError: 419911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11980.302558] LustreError: 420157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695047 with bad export cookie 898417594183306703 [11980.305092] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11980.309982] LustreError: 420157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11980.476651] Lustre: server umount lustre-MDT0001 complete [11986.417426] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11988.088383] Key type lgssc unregistered [11988.273729] LNet: 421197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.282640] LNetError: 421197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.292378] LNet: Removed LNI 192.168.204.107@tcp [11988.777673] Key type .llcrypt unregistered [11988.779501] Key type ._llcrypt unregistered [12003.721774] Key type ._llcrypt registered [12003.723925] Key type .llcrypt registered [12003.781400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12004.285495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12004.306383] alg: No test for adler32 (adler32-zlib) [12005.213645] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12005.401969] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12007.008894] Key type lgssc registered [12007.761735] Lustre: Echo OBD driver; http://www.lustre.org/ [12012.589833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12012.602855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12013.827984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12015.941970] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12020.729873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.934497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12023.704995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12028.544542] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12032.110432] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12036.623926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12036.893245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12038.958186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12040.294314] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12044.281508] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12051.430194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12051.433469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12051.437951] Lustre: Skipped 2 previous similar messages [12051.446101] Lustre: Skipped 2 previous similar messages [12054.225950] Lustre: server umount lustre-MDT0000 complete [12055.417286] LustreError: 422568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1789695122 with bad export cookie 13269968746871550575 [12055.425372] LustreError: 422568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12056.435756] LustreError: 422570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695123 with bad export cookie 13269968746871549686 [12056.440039] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12056.442675] LustreError: 422570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12056.545269] LustreError: 422589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.545296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12056.545533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12056.546329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12056.546334] Lustre: Skipped 1 previous similar message [12056.552498] LustreError: 422589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12061.666869] LustreError: 423316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.668298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12061.683269] LustreError: 423316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12062.590964] Lustre: server umount lustre-OST0000 complete [12065.763190] LustreError: 422590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.777675] LustreError: 422590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12066.626939] Lustre: server umount lustre-MDT0001 complete [12072.987567] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12074.612613] Key type lgssc unregistered [12074.810543] LNet: 426274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12074.817954] LNetError: 426274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12074.826955] LNet: Removed LNI 192.168.204.107@tcp [12075.397384] Key type .llcrypt unregistered [12075.399499] Key type ._llcrypt unregistered [12091.632178] Key type ._llcrypt registered [12091.634848] Key type .llcrypt registered [12091.713617] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12092.146903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12092.156819] alg: No test for adler32 (adler32-zlib) [12093.075551] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12093.199469] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12094.800287] Key type lgssc registered [12095.485923] Lustre: Echo OBD driver; http://www.lustre.org/ [12100.485631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12100.501848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12101.783883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12104.387477] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12109.544046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12109.753265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12112.114500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12116.354759] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12119.448035] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12123.618106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12123.855829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12126.958232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12127.718935] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12131.992456] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12137.441864] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12137.442220] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.445108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12137.463706] Lustre: Skipped 1 previous similar message [12138.784378] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [12138.789539] Lustre: Skipped 1 previous similar message [12140.775944] Lustre: server umount lustre-MDT0001 complete [12142.564749] LustreError: 427667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.575379] LustreError: 427667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12147.682311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12147.683912] LustreError: 428394:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12147.684413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12147.685622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.752946] Lustre: server umount lustre-OST0000 complete [12152.043561] Lustre: server umount lustre-MDT0000 complete [12159.540335] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12161.157323] Key type lgssc unregistered [12161.336360] LNet: 431345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12161.340778] LNetError: 431345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12161.354752] LNet: Removed LNI 192.168.204.107@tcp [12161.842107] Key type .llcrypt unregistered [12161.843663] Key type ._llcrypt unregistered [12174.980044] Key type ._llcrypt registered [12174.981892] Key type .llcrypt registered [12175.050821] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [12182.782871] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12183.216666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12183.261527] alg: No test for adler32 (adler32-zlib) [12184.207043] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12184.377656] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12185.992245] Key type lgssc registered [12186.691715] Lustre: Echo OBD driver; http://www.lustre.org/ [12208.804903] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12217.303961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12217.320470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12218.451681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12218.472340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12218.522060] Lustre: lustre-MDT0000: new disk, initializing [12218.574513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12218.592601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12220.794619] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12229.707442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12229.866709] Lustre: lustre-OST0000: new disk, initializing [12229.869994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12229.873713] Lustre: Skipped 1 previous similar message [12229.877861] Lustre: 436347:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12229.931170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12233.336613] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12234.795824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12234.802779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12234.821302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12239.517848] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12249.595184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12249.688351] Lustre: 435329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12249.737338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12249.791698] Lustre: lustre-MDT0001: new disk, initializing [12249.851019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12249.883874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12249.890405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12252.261325] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12257.577256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12257.695056] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12259.754254] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12259.882512] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12266.283061] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [12266.981377] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12266.981768] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12266.999066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12267.003628] Lustre: Skipped 1 previous similar message [12268.622537] Lustre: server umount lustre-MDT0001 complete [12277.218898] LustreError: 437058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.226367] LustreError: 437058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12282.340093] LustreError: 435336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.350660] LustreError: 435336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12285.921671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12285.924792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12285.930188] Lustre: Skipped 1 previous similar message [12285.933274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12287.458532] LustreError: 437687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12292.577954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12292.579201] LustreError: 435340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12292.582855] Lustre: Skipped 1 previous similar message [12297.184399] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12297.265996] Lustre: server umount lustre-OST0000 complete [12297.698637] LustreError: 437687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12305.455167] Lustre: server umount lustre-MDT0000 complete [12312.445551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12313.881674] Key type lgssc unregistered [12314.049485] LNet: 440092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12314.055957] LNetError: 440092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12314.066377] LNet: Removed LNI 192.168.204.107@tcp [12314.489925] Key type .llcrypt unregistered [12314.491355] Key type ._llcrypt unregistered [12335.450423] Key type ._llcrypt registered [12335.453197] Key type .llcrypt registered [12335.507356] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12335.655573] Key type .llcrypt unregistered [12335.657698] Key type ._llcrypt unregistered [12344.662268] Key type ._llcrypt registered [12344.664026] Key type .llcrypt registered [12344.734101] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12345.132778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12345.189032] alg: No test for adler32 (adler32-zlib) [12346.098156] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [12346.231132] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12347.848564] Key type lgssc registered [12348.344231] Lustre: Echo OBD driver; http://www.lustre.org/ [12362.233646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12362.254585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12363.369202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12363.384779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12363.428260] Lustre: lustre-MDT0000: new disk, initializing [12363.481964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12363.491089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12365.758742] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12371.784691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12371.859352] Lustre: 443436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12371.904559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12371.908145] Lustre: Skipped 1 previous similar message [12371.972351] Lustre: lustre-MDT0001: new disk, initializing [12372.031874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12372.053904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12372.059228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12374.180407] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12379.722427] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12382.689704] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12385.921721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12386.053733] Lustre: lustre-OST0000: new disk, initializing [12386.056646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12386.060567] Lustre: 445406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12386.106524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12387.719155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12387.724621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12387.800771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12389.219371] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12394.783481] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12398.050549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.054435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.056321] Lustre: Skipped 1 previous similar message [12398.061944] Lustre: Skipped 2 previous similar messages [12403.170859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12403.174353] Lustre: Skipped 2 previous similar messages [12404.220453] Lustre: server umount lustre-MDT0000 complete [12406.069804] LustreError: 443426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695473 with bad export cookie 10991277015783089385 [12406.073094] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12406.077297] LustreError: 443426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12406.233219] Lustre: server umount lustre-MDT0001 complete [12418.371535] Lustre: server umount lustre-OST0000 complete [12424.042503] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 21:38:11 (1789695491) [12428.071275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12428.354292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12430.596251] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12433.379053] LustreError: 447430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12433.389810] LustreError: 447430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12435.274373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12435.471911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12437.584182] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12441.695292] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12444.633777] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12461.478554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12461.599771] Lustre: Found index 0 for lustre-OST0000, updating log [12461.706397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12464.872307] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12468.842229] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12474.029828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12479.460975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12479.462318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12479.470457] Lustre: Skipped 1 previous similar message [12479.473437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12481.507768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12481.513195] Lustre: Skipped 1 previous similar message [12482.179211] Lustre: server umount lustre-OST0000 complete [12496.865760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12496.871261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.895825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12509.155878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12509.161964] Lustre: Skipped 7 previous similar messages [12510.176524] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12510.257258] Lustre: server umount lustre-MDT0000 complete [12511.967245] LustreError: 447410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789695579 with bad export cookie 10991277015783090778 [12511.968966] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12511.974343] LustreError: 447410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12512.127871] Lustre: server umount lustre-MDT0001 complete [12516.222867] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 21:39:43 (1789695583) [12532.640111] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13119.251429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13119.683726] LustreError: 453831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13119.738955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13123.674377] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13125.093330] LustreError: 453832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13130.210423] LustreError: 453831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13131.088591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13131.481558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13135.175881] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13140.865402] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13145.046601] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13162.732355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13162.904476] Lustre: Found index 0 for lustre-OST0000, updating log [13163.111796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13167.579724] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13168.236392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13172.642165] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13173.188615] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13177.765392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13177.776104] Lustre: Skipped 1 previous similar message [13177.779569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13180.825859] Lustre: server umount lustre-OST0000 complete [13184.484130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13184.488643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13184.489628] Lustre: Skipped 1 previous similar message [13184.510448] Lustre: Skipped 1 previous similar message [13188.581307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13188.587530] Lustre: Skipped 7 previous similar messages [13189.754196] Lustre: server umount lustre-MDT0000 complete [13190.627710] LustreError: 453832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.643457] LustreError: 453832:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13192.337140] LustreError: 453812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789696259 with bad export cookie 10991277015783091842 [13192.343223] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13192.345333] LustreError: 453812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13192.542688] Lustre: server umount lustre-MDT0001 complete [13217.815797] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13225.343912] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13236.129155] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13253.088816] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:52:00 (1789696320) [13266.388738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13268.528503] Key type lgssc unregistered [13268.768665] LNet: 460448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13268.774292] LNetError: 460448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13383.466471] LNet: Removed LNI 192.168.204.107@tcp [13384.126053] Key type .llcrypt unregistered [13384.127985] Key type ._llcrypt unregistered [13394.449749] Key type ._llcrypt registered [13394.451517] Key type .llcrypt registered [13394.516359] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13395.064224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13395.144272] alg: No test for adler32 (adler32-zlib) [13396.118266] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [13396.303595] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13397.952449] Key type lgssc registered [13398.868161] Lustre: Echo OBD driver; http://www.lustre.org/ [13402.380880] LNet: Added LNI 192.168.204.107@tcp100 [8/256/0/180] [13403.088954] LNet: Added LNI 192.168.204.107@tcp101 [8/256/0/180] [13403.828688] LNet: Added LNI 192.168.204.107@tcp102 [8/256/0/180] [13404.523875] LNet: Added LNI 192.168.204.107@tcp103 [8/256/0/180] [13405.186724] LNet: Added LNI 192.168.204.107@tcp104 [8/256/0/180] [13405.926883] LNet: Added LNI 192.168.204.107@tcp105 [8/256/0/180] [13406.609598] LNet: Added LNI 192.168.204.107@tcp106 [8/256/0/180] [13407.297079] LNet: Added LNI 192.168.204.107@tcp107 [8/256/0/180] [13407.964297] LNet: Added LNI 192.168.204.107@tcp108 [8/256/0/180] [13408.649922] LNet: Added LNI 192.168.204.107@tcp109 [8/256/0/180] [13409.298890] LNet: Added LNI 192.168.204.107@tcp110 [8/256/0/180] [13410.028406] LNet: Added LNI 192.168.204.107@tcp111 [8/256/0/180] [13410.809718] LNet: Added LNI 192.168.204.107@tcp112 [8/256/0/180] [13411.432856] LNet: Added LNI 192.168.204.107@tcp113 [8/256/0/180] [13412.132103] LNet: Added LNI 192.168.204.107@tcp114 [8/256/0/180] [13412.759773] LNet: Added LNI 192.168.204.107@tcp115 [8/256/0/180] [13413.504817] LNet: Added LNI 192.168.204.107@tcp116 [8/256/0/180] [13414.232070] LNet: Added LNI 192.168.204.107@tcp117 [8/256/0/180] [13414.898682] LNet: Added LNI 192.168.204.107@tcp118 [8/256/0/180] [13415.647184] LNet: Added LNI 192.168.204.107@tcp119 [8/256/0/180] [13416.367911] LNet: Added LNI 192.168.204.107@tcp120 [8/256/0/180] [13417.077668] LNet: Added LNI 192.168.204.107@tcp121 [8/256/0/180] [13417.811615] LNet: Added LNI 192.168.204.107@tcp122 [8/256/0/180] [13418.547671] LNet: Added LNI 192.168.204.107@tcp123 [8/256/0/180] [13419.370789] LNet: Added LNI 192.168.204.107@tcp124 [8/256/0/180] [13420.175321] LNet: Added LNI 192.168.204.107@tcp125 [8/256/0/180] [13420.921653] LNet: Added LNI 192.168.204.107@tcp126 [8/256/0/180] [13421.562321] LNet: Added LNI 192.168.204.107@tcp127 [8/256/0/180] [13422.245827] LNet: Added LNI 192.168.204.107@tcp128 [8/256/0/180] [13422.894688] LNet: Added LNI 192.168.204.107@tcp129 [8/256/0/180] [13423.640362] LNet: Added LNI 192.168.204.107@tcp130 [8/256/0/180] [13424.345949] LNet: Added LNI 192.168.204.107@tcp131 [8/256/0/180] [13425.059726] LNet: Added LNI 192.168.204.107@tcp132 [8/256/0/180] [13425.737663] LNet: Added LNI 192.168.204.107@tcp133 [8/256/0/180] [13426.359301] LNet: Added LNI 192.168.204.107@tcp134 [8/256/0/180] [13427.052989] LNet: Added LNI 192.168.204.107@tcp135 [8/256/0/180] [13431.811736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13431.837513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13432.928469] Lustre: 463452:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13432.970668] LNetError: 460910:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13432.979800] LNetError: Protocol error connecting to 192.168.204.107@tcp115 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13432.987659] LNetError: Skipped 3 previous similar messages [13433.116519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13433.149651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13433.184571] LNet: 8 local NIs in recovery (showing 8): 192.168.204.107@tcp122, 192.168.204.107@tcp124, 192.168.204.107@tcp125, 192.168.204.107@tcp126, 192.168.204.107@tcp127, 192.168.204.107@tcp128, 192.168.204.107@tcp129, 192.168.204.107@tcp123 [13433.240625] Lustre: lustre-MDT0000: new disk, initializing [13433.302283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13433.316764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13435.299027] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13435.308225] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [13435.317431] LNetError: Protocol error connecting to 192.168.204.107@tcp126 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13435.326828] LNetError: Skipped 15 previous similar messages [13436.458574] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13439.393651] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13439.401798] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13439.411241] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13439.420782] LNetError: Skipped 1 previous similar message [13444.548220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13444.683123] Lustre: 463471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13444.748785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13444.753642] Lustre: Skipped 1 previous similar message [13444.832701] Lustre: lustre-MDT0001: new disk, initializing [13444.877453] Lustre: 464273:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13444.884674] Lustre: 464273:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13444.983131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13445.019102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13445.024013] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13447.588566] LNetError: 460909:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13447.598228] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13448.419725] Lustre: 464505:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13448.425737] Lustre: 464505:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13448.659308] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13456.313492] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13460.649964] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13463.977737] LNetError: 460910:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13463.985656] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13466.037093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13466.297976] Lustre: lustre-OST0000: new disk, initializing [13466.301494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13466.306986] Lustre: 465445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13466.388546] Lustre: 465450:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13466.393843] Lustre: 465450:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13466.437234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13470.671569] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13472.098047] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13472.106085] LNetError: 460911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13472.113617] LNetError: Protocol error connecting to 192.168.204.107@tcp129 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13472.123969] LNetError: Skipped 7 previous similar messages [13472.287931] Lustre: 465901:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13472.302100] Lustre: 465901:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13472.357567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13472.363759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13472.476674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13478.059431] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13488.097312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13488.100377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13488.107597] LustreError: Skipped 1 previous similar message [13488.130513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13488.136622] Lustre: Skipped 1 previous similar message [13490.531368] LNetError: 460910:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13490.541820] LNetError: 460910:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [13490.549716] LNetError: Protocol error connecting to 192.168.204.107@tcp120 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13490.558183] LNetError: Skipped 9 previous similar messages [13492.640287] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp124 [13493.225696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13493.234822] Lustre: Skipped 1 previous similar message [13497.312345] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13497.459252] Lustre: server umount lustre-OST0000 complete [13501.411565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13501.417065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13501.426067] Lustre: Skipped 1 previous similar message [13501.431423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13503.463110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13503.465863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13506.798566] Lustre: server umount lustre-MDT0000 complete [13509.684339] LustreError: 463461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789696577 with bad export cookie 11526509702852925641 [13509.690915] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13509.699309] LustreError: 463461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13509.932671] Lustre: server umount lustre-MDT0001 complete [13516.873264] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13518.882215] Key type lgssc unregistered [13519.194248] LNet: 467356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13519.202986] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13520.225170] LNet: Removed LNI 192.168.204.107@tcp [13520.227839] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13521.249540] LNet: Removed LNI 192.168.204.107@tcp100 [13521.255195] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13521.261977] LNet: Removed LNI 192.168.204.107@tcp101 [13522.272232] LNet: Removed LNI 192.168.204.107@tcp102 [13523.296422] LNet: Removed LNI 192.168.204.107@tcp103 [13523.299537] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13523.307245] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13523.313966] LNet: Removed LNI 192.168.204.107@tcp104 [13524.321910] LNet: Removed LNI 192.168.204.107@tcp105 [13525.344275] LNet: Removed LNI 192.168.204.107@tcp106 [13526.368837] LNet: Removed LNI 192.168.204.107@tcp107 [13527.392384] LNet: Removed LNI 192.168.204.107@tcp108 [13527.397660] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13527.404181] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13527.409924] LNet: Removed LNI 192.168.204.107@tcp109 [13528.416219] LNet: Removed LNI 192.168.204.107@tcp110 [13529.440806] LNet: Removed LNI 192.168.204.107@tcp111 [13530.464172] LNet: Removed LNI 192.168.204.107@tcp112 [13531.488240] LNet: Removed LNI 192.168.204.107@tcp113 [13532.512218] LNet: Removed LNI 192.168.204.107@tcp114 [13533.536215] LNet: Removed LNI 192.168.204.107@tcp115 [13534.560136] LNet: Removed LNI 192.168.204.107@tcp116 [13535.584901] LNet: Removed LNI 192.168.204.107@tcp117 [13535.588082] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13535.593198] LNetError: 467356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13535.600636] LNet: Removed LNI 192.168.204.107@tcp118 [13536.608851] LNet: Removed LNI 192.168.204.107@tcp119 [13537.632240] LNet: Removed LNI 192.168.204.107@tcp120 [13538.656168] LNet: Removed LNI 192.168.204.107@tcp121 [13539.680204] LNet: Removed LNI 192.168.204.107@tcp122 [13540.704871] LNet: Removed LNI 192.168.204.107@tcp123 [13540.714603] LNet: Removed LNI 192.168.204.107@tcp124 [13541.728362] LNet: Removed LNI 192.168.204.107@tcp125 [13542.752498] LNet: Removed LNI 192.168.204.107@tcp126 [13543.777591] LNet: Removed LNI 192.168.204.107@tcp127 [13544.800204] LNet: Removed LNI 192.168.204.107@tcp128 [13545.824177] LNet: Removed LNI 192.168.204.107@tcp129 [13545.828800] LNet: Removed LNI 192.168.204.107@tcp130 [13545.833146] LNet: Removed LNI 192.168.204.107@tcp131 [13545.838055] LNet: Removed LNI 192.168.204.107@tcp132 [13545.841263] LNet: Removed LNI 192.168.204.107@tcp133 [13545.843558] LNet: Removed LNI 192.168.204.107@tcp134 [13545.857404] LNet: Removed LNI 192.168.204.107@tcp135 [13546.558063] Key type .llcrypt unregistered [13546.563410] Key type ._llcrypt unregistered [13564.873522] Key type ._llcrypt registered [13564.878938] Key type .llcrypt registered [13564.967292] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13576.182254] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13577.112698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13577.227114] alg: No test for adler32 (adler32-zlib) [13578.257417] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [13578.516262] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13580.200232] Key type lgssc registered [13581.224779] Lustre: Echo OBD driver; http://www.lustre.org/ [13608.322353] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:57:55 (1789696675) [13620.888179] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13623.132561] Key type lgssc unregistered [13623.372600] LNet: 471413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13623.383613] LNetError: 471413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13623.404050] LNet: Removed LNI 192.168.204.107@tcp [13624.119546] Key type .llcrypt unregistered [13624.123486] Key type ._llcrypt unregistered [13634.938771] Key type ._llcrypt registered [13634.940667] Key type .llcrypt registered [13635.016672] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13635.541413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13635.635450] alg: No test for adler32 (adler32-zlib) [13636.621896] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [13636.789975] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13638.435179] Key type lgssc registered [13639.237607] Lustre: Echo OBD driver; http://www.lustre.org/ [13646.274897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13646.299604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13647.498989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13647.531291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13647.612242] Lustre: lustre-MDT0000: new disk, initializing [13647.689682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13647.710093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13650.964464] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13659.135723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13659.230540] Lustre: 472601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13659.284400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13659.289421] Lustre: Skipped 1 previous similar message [13659.370391] Lustre: lustre-MDT0001: new disk, initializing [13659.444280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13659.478098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13659.485097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13662.680688] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13669.219142] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13672.563579] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13677.017804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13677.189806] Lustre: lustre-OST0000: new disk, initializing [13677.193842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13677.198821] Lustre: 474569:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13677.258404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13680.820517] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13686.332970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13686.339542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13686.400685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13687.360769] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13689.865141] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lnet_if_list [13693.804920] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13693.825752] LNet: Added LNI 192.168.204.107@tcp10 [8/256/0/180] [13697.892757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13697.921562] LNet: Added LNI 192.168.204.107@tcp11 [8/256/0/180] [13700.898466] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13700.920187] LNet: Added LNI 192.168.204.107@tcp12 [8/256/0/180] [13703.912288] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13703.935510] LNet: Added LNI 192.168.204.107@tcp13 [8/256/0/180] [13706.957516] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13706.976966] LNet: Added LNI 192.168.204.107@tcp14 [8/256/0/180] [13709.931581] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13709.958949] LNet: Added LNI 192.168.204.107@tcp15 [8/256/0/180] [13713.200938] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13713.225944] LNet: Added LNI 192.168.204.107@tcp16 [8/256/0/180] [13716.759556] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13716.794301] LNet: Added LNI 192.168.204.107@tcp17 [8/256/0/180] [13719.908785] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13719.926647] LNet: Added LNI 192.168.204.107@tcp18 [8/256/0/180] [13723.095487] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13723.118739] LNet: Added LNI 192.168.204.107@tcp19 [8/256/0/180] [13726.736605] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13726.760217] LNet: Added LNI 192.168.204.107@tcp20 [8/256/0/180] [13729.862705] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13729.883781] LNet: Added LNI 192.168.204.107@tcp21 [8/256/0/180] [13733.364726] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13733.384455] LNet: Added LNI 192.168.204.107@tcp22 [8/256/0/180] [13737.110948] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13737.133715] LNet: Added LNI 192.168.204.107@tcp23 [8/256/0/180] [13740.735892] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13740.767769] LNet: Added LNI 192.168.204.107@tcp24 [8/256/0/180] [13744.696490] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13744.724168] LNet: Added LNI 192.168.204.107@tcp25 [8/256/0/180] [13748.608239] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13748.626047] LNet: Added LNI 192.168.204.107@tcp26 [8/256/0/180] [13752.734989] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13752.760798] LNet: Added LNI 192.168.204.107@tcp27 [8/256/0/180] [13757.067890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13757.096130] LNet: Added LNI 192.168.204.107@tcp28 [8/256/0/180] [13761.464688] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13761.496109] LNet: Added LNI 192.168.204.107@tcp29 [8/256/0/180] [13765.981986] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13766.010364] LNet: Added LNI 192.168.204.107@tcp30 [8/256/0/180] [13770.622841] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13770.647945] LNet: Added LNI 192.168.204.107@tcp31 [8/256/0/180] [13774.502922] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13774.524912] LNet: Added LNI 192.168.204.107@tcp32 [8/256/0/180] [13778.760025] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13778.776446] LNet: Added LNI 192.168.204.107@tcp33 [8/256/0/180] [13782.878731] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13782.906877] LNet: Added LNI 192.168.204.107@tcp34 [8/256/0/180] [13786.963893] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13786.990193] LNet: Added LNI 192.168.204.107@tcp35 [8/256/0/180] [13791.463129] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13791.489434] LNet: Added LNI 192.168.204.107@tcp36 [8/256/0/180] [13796.094862] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13796.116460] LNet: Added LNI 192.168.204.107@tcp37 [8/256/0/180] [13800.541462] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13800.561191] LNet: Added LNI 192.168.204.107@tcp38 [8/256/0/180] [13805.385988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13805.412676] LNet: Added LNI 192.168.204.107@tcp39 [8/256/0/180] [13810.320115] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13810.341191] LNet: Added LNI 192.168.204.107@tcp40 [8/256/0/180] [13815.264850] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13815.286342] LNet: Added LNI 192.168.204.107@tcp41 [8/256/0/180] [13820.251236] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13820.282549] LNet: Added LNI 192.168.204.107@tcp42 [8/256/0/180] [13825.436459] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13825.459600] LNet: Added LNI 192.168.204.107@tcp43 [8/256/0/180] [13830.908514] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13830.935664] LNet: Added LNI 192.168.204.107@tcp44 [8/256/0/180] [13836.370535] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13836.390645] LNet: Added LNI 192.168.204.107@tcp45 [8/256/0/180] [13841.582750] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13841.603476] LNet: Added LNI 192.168.204.107@tcp46 [8/256/0/180] [13846.680622] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13846.704634] LNet: Added LNI 192.168.204.107@tcp47 [8/256/0/180] [13851.701894] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13851.722914] LNet: Added LNI 192.168.204.107@tcp48 [8/256/0/180] [13857.130538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13857.162207] LNet: Added LNI 192.168.204.107@tcp49 [8/256/0/180] [13862.412868] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13862.432492] LNet: Added LNI 192.168.204.107@tcp50 [8/256/0/180] [13874.659028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13874.664950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13874.676866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13876.194433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13876.199966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13876.202676] Lustre: Skipped 1 previous similar message [13876.212409] Lustre: Skipped 1 previous similar message [13879.839869] Lustre: server umount lustre-MDT0000 complete [13882.036659] LustreError: 472594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789696949 with bad export cookie 7362554645688800274 [13882.040178] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13882.045137] LustreError: 472594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13882.249304] Lustre: server umount lustre-MDT0001 complete [13893.977326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13899.551753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13901.792158] Lustre: 471876:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789696953/real 1789696953] req@00000000368d6a15 x1876633011216000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789696969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13901.818170] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13902.816256] Lustre: 471875:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789696954/real 1789696954] req@000000008fe7e558 x1876633011216128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789696970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13906.912873] Lustre: 471874:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789696958/real 1789696958] req@000000001ef783b5 x1876633011216256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789696974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13906.976693] LustreError: 471872:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000074f73710 x1876633011217024/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13907.131989] LustreError: 474563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13907.145921] LustreError: 474563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13907.172863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13909.821930] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13910.178713] LustreError: 474564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13912.233494] LustreError: 498782:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13912.245321] LustreError: 498782:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13914.272416] Lustre: 471875:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789696965/real 1789696965] req@000000009b873532 x1876633011216640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789696981 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13914.294971] Lustre: 471875:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13915.080622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13915.373328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13917.865531] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13920.743230] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13921.876711] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13925.116176] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13930.988167] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13936.101322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13936.102493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13936.112749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13936.122170] Lustre: Skipped 2 previous similar messages [13941.219801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13941.221791] Lustre: Skipped 1 previous similar message [13942.053570] Lustre: server umount lustre-OST0000 complete [13946.340421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13946.341961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13946.352532] Lustre: Skipped 1 previous similar message [13946.358963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13946.367695] Lustre: Skipped 1 previous similar message [13950.271026] Lustre: server umount lustre-MDT0000 complete [13951.457438] LustreError: 498785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13951.466044] LustreError: 498785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13952.388520] LustreError: 473397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789697019 with bad export cookie 7362554645688802605 [13952.392340] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13952.394508] LustreError: 473397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13952.564658] Lustre: server umount lustre-MDT0001 complete [13958.588518] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13960.342924] Key type lgssc unregistered [13960.592804] LNet: 501555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13960.598430] LNetError: 501555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13960.602861] LNet: Removed LNI 192.168.204.107@tcp [13960.607525] LNet: Removed LNI 192.168.204.107@tcp10 [13960.610246] LNet: Removed LNI 192.168.204.107@tcp11 [13960.612311] LNet: Removed LNI 192.168.204.107@tcp12 [13960.615618] LNet: Removed LNI 192.168.204.107@tcp13 [13960.617367] LNet: Removed LNI 192.168.204.107@tcp14 [13960.620332] LNet: Removed LNI 192.168.204.107@tcp15 [13960.622803] LNet: Removed LNI 192.168.204.107@tcp16 [13960.625760] LNet: Removed LNI 192.168.204.107@tcp17 [13960.629574] LNet: Removed LNI 192.168.204.107@tcp18 [13960.632555] LNet: Removed LNI 192.168.204.107@tcp19 [13960.635366] LNet: Removed LNI 192.168.204.107@tcp20 [13960.637247] LNet: Removed LNI 192.168.204.107@tcp21 [13960.640559] LNet: Removed LNI 192.168.204.107@tcp22 [13960.643330] LNet: Removed LNI 192.168.204.107@tcp23 [13960.645557] LNet: Removed LNI 192.168.204.107@tcp24 [13960.648202] LNet: Removed LNI 192.168.204.107@tcp25 [13960.650416] LNet: Removed LNI 192.168.204.107@tcp26 [13960.652278] LNet: Removed LNI 192.168.204.107@tcp27 [13960.656611] LNet: Removed LNI 192.168.204.107@tcp28 [13960.659321] LNet: Removed LNI 192.168.204.107@tcp29 [13960.662939] LNet: Removed LNI 192.168.204.107@tcp30 [13960.665469] LNet: Removed LNI 192.168.204.107@tcp31 [13960.667735] LNet: Removed LNI 192.168.204.107@tcp32 [13960.670372] LNet: Removed LNI 192.168.204.107@tcp33 [13960.672656] LNet: Removed LNI 192.168.204.107@tcp34 [13960.675393] LNet: Removed LNI 192.168.204.107@tcp35 [13960.677685] LNet: Removed LNI 192.168.204.107@tcp36 [13960.681978] LNet: Removed LNI 192.168.204.107@tcp37 [13960.684296] LNet: Removed LNI 192.168.204.107@tcp38 [13960.689537] LNet: Removed LNI 192.168.204.107@tcp39 [13960.691767] LNet: Removed LNI 192.168.204.107@tcp40 [13960.694873] LNet: Removed LNI 192.168.204.107@tcp41 [13960.697050] LNet: Removed LNI 192.168.204.107@tcp42 [13960.698940] LNet: Removed LNI 192.168.204.107@tcp43 [13960.701586] LNet: Removed LNI 192.168.204.107@tcp44 [13960.703539] LNet: Removed LNI 192.168.204.107@tcp45 [13960.705459] LNet: Removed LNI 192.168.204.107@tcp46 [13960.707842] LNet: Removed LNI 192.168.204.107@tcp47 [13960.710247] LNet: Removed LNI 192.168.204.107@tcp48 [13960.713236] LNet: Removed LNI 192.168.204.107@tcp49 [13960.723196] LNet: Removed LNI 192.168.204.107@tcp50 [13961.202759] Key type .llcrypt unregistered [13961.205379] Key type ._llcrypt unregistered [13972.861300] Key type ._llcrypt registered [13972.864530] Key type .llcrypt registered [13972.949706] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13982.961091] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13983.600506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13983.622316] alg: No test for adler32 (adler32-zlib) [13984.550115] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [13984.702784] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13986.320385] Key type lgssc registered [13987.015083] Lustre: Echo OBD driver; http://www.lustre.org/ [14007.993314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14008.012885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.152676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14009.176932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14009.230598] Lustre: lustre-MDT0000: new disk, initializing [14009.285414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14009.298192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14011.839761] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14017.946142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14018.013450] Lustre: 504722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14018.050492] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14018.053196] Lustre: Skipped 1 previous similar message [14018.103463] Lustre: lustre-MDT0001: new disk, initializing [14018.171445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14018.195188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14018.199954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14020.655648] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14026.456171] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14029.897933] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14034.305095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14034.436815] Lustre: *** cfs_fail_loc=90f, val=0*** [14034.485650] Lustre: lustre-OST0000: new disk, initializing [14034.489467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14034.494858] Lustre: 506739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14034.555727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14038.438571] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14044.657776] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14044.690158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14044.696387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14044.771354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14049.762273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14049.763903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14049.770883] LustreError: Skipped 1 previous similar message [14049.772792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14049.777712] Lustre: Skipped 1 previous similar message [14049.789044] Lustre: Skipped 1 previous similar message [14051.087551] Lustre: server umount lustre-OST0000 complete [14054.369416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14054.375192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14054.386504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14059.588245] Lustre: server umount lustre-MDT0000 complete [14062.057767] LustreError: 505517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789697129 with bad export cookie 1337879994485456126 [14062.060262] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14062.067949] LustreError: 505517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14062.273430] Lustre: server umount lustre-MDT0001 complete [14069.550717] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14071.609778] Key type lgssc unregistered [14071.842794] LNet: 508640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14071.849264] LNetError: 508640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14071.866479] LNet: Removed LNI 192.168.204.107@tcp [14072.537681] Key type .llcrypt unregistered [14072.540319] Key type ._llcrypt unregistered [14085.050901] Key type ._llcrypt registered [14085.053393] Key type .llcrypt registered [14085.135523] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14085.618477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14085.628376] alg: No test for adler32 (adler32-zlib) [14086.536881] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [14086.680509] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14088.305462] Key type lgssc registered [14089.101741] Lustre: Echo OBD driver; http://www.lustre.org/ [14093.852165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14093.878836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14095.184050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14098.187988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14103.595569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14103.806238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14106.360543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14111.092312] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14114.790887] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14118.708248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14118.913539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14122.276052] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14126.864893] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14134.552190] loop0: detected capacity change from 0 to 204800000 [14134.675377] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14134.684515] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14134.695168] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14138.054129] loop0: detected capacity change from 0 to 204800000 [14138.190081] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14138.197983] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14138.207516] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14141.265070] loop0: detected capacity change from 0 to 204800000 [14144.510084] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14144.605478] Lustre: fs15246-OST0000: new disk, initializing [14144.609411] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14144.615356] Lustre: 513474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14144.657197] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14148.249868] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14154.237634] loop1: detected capacity change from 0 to 204800000 [14157.711738] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14157.769989] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14157.820719] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14157.889736] Lustre: fs15246-MDT0000: new disk, initializing [14157.995638] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14158.011723] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14158.067216] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14159.075898] Lustre: 514596:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009b8a5975 x1876633482989440/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789697242 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14160.640209] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14168.164631] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14168.169453] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14168.190879] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14176.233568] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14176.246203] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14179.636157] Lustre: server umount fs15246-MDT0000 complete [14181.357331] LustreError: 509882:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14181.378344] LustreError: 509882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14186.350092] Lustre: server umount fs15246-OST0000 complete [14195.681763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14195.687663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14195.704833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14196.705225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14196.710579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14196.711065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14200.228583] Lustre: server umount lustre-OST0000 complete [14206.434771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14206.444186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14206.464445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14206.467683] Lustre: Skipped 1 previous similar message [14209.235658] Lustre: server umount lustre-MDT0000 complete [14211.460410] LustreError: 509866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789697278 with bad export cookie 6985903640848149143 [14211.462110] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14211.468516] LustreError: 509866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14211.691829] Lustre: server umount lustre-MDT0001 complete [14218.120801] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14219.994808] Key type lgssc unregistered [14220.210706] LNet: 517686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14220.216890] LNetError: 517686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14220.245244] LNet: Removed LNI 192.168.204.107@tcp [14220.838277] Key type .llcrypt unregistered [14220.840339] Key type ._llcrypt unregistered [14232.385970] Key type ._llcrypt registered [14232.387496] Key type .llcrypt registered [14232.449715] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14233.040088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14233.105401] alg: No test for adler32 (adler32-zlib) [14234.119381] Lustre: Lustre: Build Version: 2.17.58_39_g12a437a [14234.299649] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14235.944217] Key type lgssc registered [14236.567857] Lustre: Echo OBD driver; http://www.lustre.org/ [14260.834738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14266.374742] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14288.210859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14288.228924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14289.414532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14289.432977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14289.487714] Lustre: lustre-MDT0000: new disk, initializing [14289.559786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14289.575164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14292.138661] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14299.463614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14299.585156] Lustre: 522122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14299.638784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14299.642309] Lustre: Skipped 1 previous similar message [14299.709887] Lustre: lustre-MDT0001: new disk, initializing [14299.802216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14299.839829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14299.851275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14302.662502] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14308.731380] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14312.307414] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14316.501202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14316.658554] Lustre: lustre-OST0000: new disk, initializing [14316.661929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14316.669597] Lustre: 524088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14316.736963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14320.079960] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14322.734609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14322.745000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14322.847187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14326.289677] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14328.649979] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14328.771678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14331.069413] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14331.180586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14333.411969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14333.413498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14333.419377] LustreError: Skipped 1 previous similar message [14333.420473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14333.429340] Lustre: Skipped 1 previous similar message [14338.511293] Lustre: server umount lustre-OST0000 complete [14341.090053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14341.094478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14341.102145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14341.106812] Lustre: Skipped 1 previous similar message [14343.655558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14343.657181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14347.150434] Lustre: server umount lustre-MDT0000 complete [14349.393323] LustreError: 522114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789697416 with bad export cookie 2540217652920414884 [14349.401318] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14349.405760] LustreError: 522114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14349.600704] Lustre: server umount lustre-MDT0001 complete [14362.904780] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14368.157104] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14384.488686] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14066 sec ========= 22:10:51 (1789697451) [14385.398860] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:10:52 (1789697452) === [14387.173423] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:10:54 (1789697454) === [14408.876983] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14410.494196] Key type lgssc unregistered [14410.689664] LNet: 530719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14410.695322] LNetError: 530719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14410.704680] LNet: Removed LNI 192.168.204.107@tcp [14411.182104] Key type .llcrypt unregistered [14411.183743] Key type ._llcrypt unregistered