[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 446708429 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002275] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008008] pid_max: default: 32768 minimum: 301 [ 0.009099] LSM: Security Framework initializing [ 0.010031] Yama: becoming mindful. [ 0.011020] SELinux: Initializing. [ 0.012041] *** VALIDATE selinux *** [ 0.018604] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022507] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023121] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026052] *** VALIDATE tmpfs *** [ 0.027063] *** VALIDATE proc *** [ 0.028036] *** VALIDATE cgroup *** [ 0.029008] *** VALIDATE cgroup2 *** [ 0.030224] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032114] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034027] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038065] debug: unmapping init [mem 0xffffffffad459000-0xffffffffad460fff] [ 0.040150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041535] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042019] ... version: 2 [ 0.042836] ... bit width: 48 [ 0.043008] ... generic registers: 4 [ 0.043732] ... value mask: 0000ffffffffffff [ 0.044010] ... max period: 00007fffffffffff [ 0.045007] ... fixed-purpose events: 3 [ 0.045664] ... event mask: 000000070000000f [ 0.046273] rcu: Hierarchical SRCU implementation. [ 0.048143] smp: Bringing up secondary CPUs ... [ 0.049398] x86: Booting SMP configuration: [ 0.050019] .... node #0, CPUs: #1 #2 #3 [ 0.056076] smp: Brought up 1 node, 4 CPUs [ 0.057975] smpboot: Max logical packages: 1 [ 0.058009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.077568] node 0 deferred pages initialised in 18ms [ 0.080075] devtmpfs: initialized [ 0.081070] x86/mm: Memory block size: 128MB [ 0.083351] gcov: version magic: 0x41383552 [ 0.084711] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.085071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.086209] pinctrl core: initialized pinctrl subsystem [ 0.087154] [ 0.087432] ************************************************************* [ 0.088009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.089007] ** ** [ 0.090007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.091008] ** ** [ 0.092007] ** This means that this kernel is built to expose internal ** [ 0.093008] ** IOMMU data structures, which may compromise security on ** [ 0.094011] ** your system. ** [ 0.095008] ** ** [ 0.096009] ** If you see this message and you are not debugging the ** [ 0.097008] ** kernel, report this immediately to your vendor! ** [ 0.098010] ** ** [ 0.099010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.100009] ************************************************************* [ 0.101653] NET: Registered protocol family 16 [ 0.102434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.103052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.104053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.105557] cpuidle: using governor menu [ 0.107463] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.110509] PCI: Using configuration type 1 for base access [ 0.112137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.121244] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.123075] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.125244] cryptd: max_cpu_qlen set to 1000 [ 0.127708] ACPI: Added _OSI(Module Device) [ 0.129014] ACPI: Added _OSI(Processor Device) [ 0.130012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.132015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.137590] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.143363] ACPI: Interpreter enabled [ 0.144040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.144985] ACPI: Using IOAPIC for interrupt routing [ 0.146186] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.149392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.159704] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.161039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.164017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.166107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.171544] acpiphp: Slot [2] registered [ 0.172098] acpiphp: Slot [5] registered [ 0.174085] acpiphp: Slot [6] registered [ 0.175121] acpiphp: Slot [7] registered [ 0.176081] acpiphp: Slot [8] registered [ 0.177140] acpiphp: Slot [9] registered [ 0.179097] acpiphp: Slot [10] registered [ 0.180097] acpiphp: Slot [3] registered [ 0.181072] acpiphp: Slot [4] registered [ 0.182135] acpiphp: Slot [11] registered [ 0.183063] acpiphp: Slot [12] registered [ 0.185068] acpiphp: Slot [13] registered [ 0.186073] acpiphp: Slot [14] registered [ 0.187140] acpiphp: Slot [15] registered [ 0.188074] acpiphp: Slot [16] registered [ 0.189073] acpiphp: Slot [17] registered [ 0.191075] acpiphp: Slot [18] registered [ 0.192085] acpiphp: Slot [19] registered [ 0.193160] acpiphp: Slot [20] registered [ 0.195080] acpiphp: Slot [21] registered [ 0.196071] acpiphp: Slot [22] registered [ 0.197055] acpiphp: Slot [23] registered [ 0.198000] acpiphp: Slot [24] registered [ 0.199072] acpiphp: Slot [25] registered [ 0.200073] acpiphp: Slot [26] registered [ 0.201189] acpiphp: Slot [27] registered [ 0.203072] acpiphp: Slot [28] registered [ 0.204071] acpiphp: Slot [29] registered [ 0.205080] acpiphp: Slot [30] registered [ 0.207095] acpiphp: Slot [31] registered [ 0.208038] PCI host bridge to bus 0000:00 [ 0.209016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.211020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.213018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.215019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.217022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.220018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.221151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.224601] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.226890] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.235859] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.241066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.243016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.247018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.249013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.251536] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.253811] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.257040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.259858] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.265015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.276000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.282018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.286823] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.292015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.298015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.316025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.324810] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.335016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.345014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.366017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.378808] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.407035] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.418023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.464028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.510046] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.523017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.536015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.568017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583642] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620044] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.652028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.753026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.797987] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.814018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.836018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.945023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.984348] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.985304] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.987332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.989267] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.991161] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.995126] iommu: Default domain type: Passthrough [ 0.997484] SCSI subsystem initialized [ 0.998121] ACPI: bus type USB registered [ 0.999075] usbcore: registered new interface driver usbfs [ 1.001097] usbcore: registered new interface driver hub [ 1.002069] usbcore: registered new device driver usb [ 1.003153] pps_core: LinuxPPS API ver. 1 registered [ 1.005007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.007048] PTP clock support registered [ 1.009123] EDAC MC: Ver: 3.0.0 [ 1.011002] PCI: Using ACPI for IRQ routing [ 1.013000] NetLabel: Initializing [ 1.013899] NetLabel: domain hash size = 128 [ 1.015008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.017087] NetLabel: unlabeled traffic allowed by default [ 1.020103] vgaarb: loaded [ 1.021306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.022013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.028263] clocksource: Switched to clocksource kvm-clock [ 1.149735] VFS: Disk quotas dquot_6.6.0 [ 1.151324] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.153196] *** VALIDATE ramfs *** [ 1.154121] *** VALIDATE hugetlbfs *** [ 1.155297] pnp: PnP ACPI init [ 1.158102] pnp: PnP ACPI: found 6 devices [ 1.173567] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.176728] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.178609] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.180402] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.182561] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.184672] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.187070] NET: Registered protocol family 2 [ 1.189503] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.194700] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.198385] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.203942] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.207048] TCP: Hash tables configured (established 65536 bind 65536) [ 1.210266] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.213736] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.217411] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.220430] NET: Registered protocol family 1 [ 1.227356] RPC: Registered named UNIX socket transport module. [ 1.229870] RPC: Registered udp transport module. [ 1.231834] RPC: Registered tcp transport module. [ 1.233535] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.235747] NET: Registered protocol family 44 [ 1.237526] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.239713] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.241860] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.244414] PCI: CLS 0 bytes, default 64 [ 1.246264] Unpacking initramfs... [ 2.647147] debug: unmapping init [mem 0xffff99a03cc54000-0xffff99a03ffbffff] [ 2.652458] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.654008] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.657303] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.216552] Initialise system trusted keyrings [ 3.217828] Key type blacklist registered [ 3.219141] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.226819] zbud: loaded [ 3.229386] *** VALIDATE nfs *** [ 3.230202] *** VALIDATE nfs4 *** [ 3.231282] pstore: using deflate compression [ 3.234144] Platform Keyring initialized [ 3.338960] NET: Registered protocol family 38 [ 3.340576] Key type asymmetric registered [ 3.341816] Asymmetric key parser 'x509' registered [ 3.343168] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.345543] io scheduler mq-deadline registered [ 3.346803] io scheduler kyber registered [ 3.348259] io scheduler bfq registered [ 3.349698] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.352116] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.354337] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.356460] ACPI: Power Button [PWRF] [ 3.360986] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.367207] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.378464] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.384667] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.397682] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.425441] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.453381] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.456967] Non-volatile memory driver v1.3 [ 3.458295] Linux agpgart interface v0.103 [ 3.484540] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.486626] vda: detected capacity change from 0 to 74694656 [ 3.499695] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.502873] vdb: detected capacity change from 0 to 1073741824 [ 3.515857] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.518602] vdc: detected capacity change from 0 to 2621440000 [ 3.532548] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.535341] vdd: detected capacity change from 0 to 2621440000 [ 3.557263] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.559980] vde: detected capacity change from 0 to 4294967296 [ 3.596649] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.599249] vdf: detected capacity change from 0 to 4294967296 [ 3.605152] libphy: Fixed MDIO Bus: probed [ 3.631044] usbcore: registered new interface driver usbserial_generic [ 3.635895] usbserial: USB Serial support registered for generic [ 3.638488] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.641144] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.642181] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.646501] mousedev: PS/2 mouse device common for all mice [ 3.650868] rtc_cmos 00:05: RTC can wake from S4 [ 3.651894] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.655348] rtc_cmos 00:05: registered as rtc0 [ 3.657262] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.658677] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.661766] intel_pstate: CPU model not supported [ 3.664052] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.666619] hid: raw HID events driver (C) Jiri Kosina [ 3.668219] usbcore: registered new interface driver usbhid [ 3.669573] usbhid: USB HID core driver [ 3.670614] drop_monitor: Initializing network drop monitor service [ 3.672257] Initializing XFRM netlink socket [ 3.673678] NET: Registered protocol family 10 [ 3.675989] Segment Routing with IPv6 [ 3.677077] NET: Registered protocol family 17 [ 3.678960] mpls_gso: MPLS GSO support [ 3.684958] RAS: Correctable Errors collector initialized. [ 3.686560] AVX version of gcm_enc/dec engaged. [ 3.687693] AES CTR mode by8 optimization enabled [ 3.760543] sched_clock: Marking stable (3760514473, 0)->(4597514982, -837000509) [ 3.763296] registered taskstats version 1 [ 3.765144] Loading compiled-in X.509 certificates [ 3.766819] zswap: loaded using pool lzo/zbud [ 3.791848] Key type big_key registered [ 3.803076] Key type encrypted registered [ 3.804241] ima: No TPM chip found, activating TPM-bypass! [ 3.805626] ima: Allocated hash algorithm: sha1 [ 3.806880] ima: No architecture policies found [ 3.808073] evm: Initialising EVM extended attributes: [ 3.809367] evm: security.selinux [ 3.810268] evm: security.ima [ 3.811103] evm: security.capability [ 3.812028] evm: HMAC attrs: 0x1 [ 3.813925] rtc_cmos 00:05: setting system clock to 2026-07-29 08:51:23 UTC (1785315083) [ 3.819229] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 3.821636] debug: unmapping init [mem 0xffffffffad182000-0xffffffffad458fff] [ 3.831077] Write protecting the kernel read-only data: 28672k [ 3.833948] debug: unmapping init [mem 0xffffffffab803000-0xffffffffab9fffff] [ 3.836209] debug: unmapping init [mem 0xffffffffac114000-0xffffffffac1fffff] [ 3.867676] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.874203] systemd[1]: Detected virtualization kvm. [ 3.875550] systemd[1]: Detected architecture x86-64. [ 3.876869] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.903227] systemd[1]: No hostname configured. [ 3.904845] systemd[1]: Set hostname to . [ 3.906478] random: systemd: uninitialized urandom read (16 bytes read) [ 3.908591] systemd[1]: Initializing machine ID from random generator. [ 4.040523] random: systemd: uninitialized urandom read (16 bytes read) [ 4.042654] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.045405] random: systemd: uninitialized urandom read (16 bytes read) [ 4.047685] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.051125] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.801771] device-mapper: uevent: version 1.0.3 [ 4.808500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.200743] random: fast init done [ 7.207442] scsi host0: ata_piix [ 7.225768] virtio_net virtio0 ens2: renamed from eth0 [ 7.319287] scsi host1: ata_piix [ 7.324470] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.330989] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.331858] random: crng init done [ 12.337080] random: 7 urandom warning(s) missed due to ratelimiting [ 15.619038] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.729971] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.202792] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.060122] SELinux: Disabled at runtime. [ 24.279899] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.308229] systemd[1]: Detected virtualization kvm. [ 24.309883] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.492356] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.497312] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.505809] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.514951] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.521416] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.530962] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.548235] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. [ 27.106308] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 28.309872] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.480740] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.576061] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 30.269653] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.491779] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 35.732533] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 37.195411] NFS: Registering the id_resolver key type [ 37.200115] Key type id_resolver registered [ 37.206317] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg108-server login: [ 98.706640] libcfs: loading out-of-tree module taints kernel. [ 98.755880] Key type ._llcrypt registered [ 98.768166] Key type .llcrypt registered [ 98.919708] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 120.761905] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 123.206612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.260541] alg: No test for adler32 (adler32-zlib) [ 125.358309] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 127.075051] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 129.225770] Key type lgssc registered [ 132.107850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 134.972208] hrtimer: interrupt took 3198885 ns [ 153.804869] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 201.174903] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 217.406164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 217.454256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.766433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 218.801889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.002134] Lustre: lustre-MDT0000: new disk, initializing [ 219.173884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.216490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 224.491589] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.684057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.856185] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 240.918145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 240.926138] Lustre: Skipped 1 previous similar message [ 241.093271] Lustre: lustre-MDT0001: new disk, initializing [ 241.217350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.272608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 241.309377] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.680525] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.672674] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 264.781166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.244400] Lustre: lustre-OST0000: new disk, initializing [ 265.258489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 265.274497] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.401356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.848113] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.494773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.511331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.600338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 292.314938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.503300] Lustre: lustre-OST0001: new disk, initializing [ 292.510484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 292.522297] Lustre: 9534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.729606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.389884] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.138945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 302.158417] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 302.237343] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 315.875707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 327.417059] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 337.582080] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing check_logdir /tmp/testlogs/ [ 344.831320] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing yml_node [ 350.474266] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 353.925648] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 357.461068] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 359.814519] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jul 29 04:57:16 EDT 2026 [ 381.817632] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 384.163281] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 404.969674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.971608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 404.972592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 404.982056] Lustre: Skipped 3 previous similar messages [ 409.222099] Lustre: server umount lustre-MDT0000 complete [ 413.858993] LustreError: 9533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315493 with bad export cookie 6035204526851097458 [ 413.861121] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.879821] LustreError: 9533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 415.200178] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.200193] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.238469] LustreError: 6528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 415.284138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.292163] Lustre: Skipped 4 previous similar messages [ 418.208767] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.208773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 420.648747] Lustre: server umount lustre-MDT0001 complete [ 426.827976] Lustre: server umount lustre-OST0000 complete [ 432.982431] Lustre: server umount lustre-OST0001 complete [ 443.525235] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 454.653756] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 505.616039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.037262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 506.098146] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 506.221120] Lustre: lustre-MDT0000: new disk, initializing [ 506.323841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.350612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 512.410567] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 529.789442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.187376] Lustre: 15094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 530.395084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 530.401617] Lustre: Skipped 1 previous similar message [ 530.775139] Lustre: lustre-MDT0001: new disk, initializing [ 530.936204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 530.986015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 531.007756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 541.209927] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 556.877682] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.282222] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 572.368280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.840078] Lustre: lustre-OST0000: new disk, initializing [ 572.846760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 572.865709] Lustre: 17064:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 573.036742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.949636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 574.960693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 575.133197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 581.684444] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 592.877952] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 598.301463] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 598.468408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 604.328780] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 604.637834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 610.820708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.821847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 610.843350] Lustre: Skipped 1 previous similar message [ 610.846756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 610.846762] Lustre: Skipped 1 previous similar message [ 613.866619] Lustre: server umount lustre-OST0000 complete [ 621.024687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.045575] Lustre: Skipped 1 previous similar message [ 621.061307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.069959] Lustre: Skipped 2 previous similar messages [ 626.148395] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.158073] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 626.229309] Lustre: server umount lustre-MDT0000 complete [ 631.264680] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.291724] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 631.341226] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315711 with bad export cookie 6035204526851098998 [ 631.343351] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.348600] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 632.329481] Lustre: server umount lustre-MDT0001 complete [ 638.354373] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:01:55 (1785315715) [ 641.426893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 643.862279] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:02:01 (1785315721) [ 653.189329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.183024] LustreError: 19488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.390527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.285450] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 663.926400] LustreError: 19489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.954791] LustreError: 19489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 674.981675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.729265] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 692.419394] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 700.504309] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 710.057949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.916658] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 727.356559] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 736.228357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 736.234580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.250303] Lustre: Skipped 1 previous similar message [ 736.267414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 739.312670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 742.385332] Lustre: server umount lustre-OST0000 complete [ 749.536347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.556718] Lustre: Skipped 2 previous similar messages [ 754.656391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.673328] Lustre: Skipped 5 previous similar messages [ 762.335939] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.969530] Lustre: server umount lustre-MDT0000 complete [ 764.898243] LustreError: 19483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.933475] LustreError: 19483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 769.216873] LustreError: 19470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785315848 with bad export cookie 6035204526851099880 [ 769.231289] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.235568] LustreError: 19470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 770.467361] Lustre: server umount lustre-MDT0001 complete [ 781.760510] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 785.888669] Key type lgssc unregistered [ 786.661283] LNet: 23030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.674119] LNetError: 23030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.711779] LNet: Removed LNI 192.168.201.108@tcp [ 788.283817] Key type .llcrypt unregistered [ 788.286438] Key type ._llcrypt unregistered [ 819.820660] Key type ._llcrypt registered [ 819.822877] Key type .llcrypt registered [ 820.024473] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 821.756799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 821.803993] alg: No test for adler32 (adler32-zlib) [ 822.896656] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 823.137556] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 824.815302] Key type lgssc registered [ 825.762970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 835.466877] loop: module loaded [ 835.539275] loop0: detected capacity change from 0 to 204800000 [ 835.863262] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 835.888166] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 836.416935] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 844.664564] loop0: detected capacity change from 0 to 204800000 [ 844.893936] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 844.905626] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 844.922715] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 850.955775] loop0: detected capacity change from 0 to 204800000 [ 858.440539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 858.474168] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.104119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 860.192157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 860.431848] Lustre: lustre-MDT0000: new disk, initializing [ 860.590352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.655057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 869.145193] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 884.904212] loop1: detected capacity change from 0 to 204800000 [ 893.791690] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 894.240056] Lustre: lustre-OST07c6: new disk, initializing [ 894.243329] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 894.249211] Lustre: Skipped 1 previous similar message [ 894.257803] Lustre: 26222:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 894.368435] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 902.188303] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 904.295085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 904.311945] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 904.367777] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 907.612781] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 907.638449] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 907.647274] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 907.666379] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 907.683482] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 907.692121] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 924.122360] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 924.141884] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 924.153179] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 924.161748] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 924.170942] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 924.183467] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 930.178570] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 930.198385] Lustre: 25206:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 930.209778] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 930.221650] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 930.230425] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 930.238198] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 930.257860] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 930.268022] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 930.280992] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 930.290311] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 930.299597] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 930.309080] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 933.555869] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 933.576824] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 933.593647] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 933.605270] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 933.623113] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 933.637946] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 933.646782] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 933.666268] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 933.680283] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 933.696293] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 933.708380] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 933.723422] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 942.234554] Lustre: 26432:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 942.246167] Lustre: 26432:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 942.253279] Lustre: 26432:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 942.272364] Lustre: 26432:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 942.282722] Lustre: 26432:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 942.293203] Lustre: 26432:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 942.305171] Lustre: 26432:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 942.315126] Lustre: 26432:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 942.324748] Lustre: 26432:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 942.341608] Lustre: 26432:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 942.353120] Lustre: 26432:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 942.365361] Lustre: 26432:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 952.149490] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 952.162102] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 952.169918] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 952.180233] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 952.193889] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 952.207977] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 952.217348] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 952.228828] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 952.240277] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 952.258871] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 952.267971] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 952.278972] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1001.441417] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.458134] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1002.059648] Lustre: server umount lustre-OST07c6 complete [ 1013.978828] Lustre: server umount lustre-MDT0000 complete [ 1038.721315] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1042.610329] Key type lgssc unregistered [ 1043.046492] LNet: 28803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.058282] LNetError: 28803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.069948] LNet: Removed LNI 192.168.201.108@tcp [ 1045.596022] Key type .llcrypt unregistered [ 1045.599535] Key type ._llcrypt unregistered [ 1065.845269] Key type ._llcrypt registered [ 1065.846795] Key type .llcrypt registered [ 1065.983352] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1067.516812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.836348] alg: No test for adler32 (adler32-zlib) [ 1068.913234] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 1069.175458] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1070.840078] Key type lgssc registered [ 1071.987713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1080.662991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1080.690315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.382563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.863506] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1092.581723] LustreError: 30075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.603184] LustreError: 30075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1097.704198] LustreError: 30076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.405359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.812424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1105.450084] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1115.116752] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1121.931389] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1129.823708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1130.369277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.565601] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1150.130564] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1152.884111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1173.474219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1173.492057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.515709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1173.527490] Lustre: Skipped 1 previous similar message [ 1178.595597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.611874] Lustre: Skipped 1 previous similar message [ 1179.712490] Lustre: server umount lustre-OST0000 complete [ 1186.787295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1186.806430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.859751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.874715] Lustre: Skipped 1 previous similar message [ 1188.835327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.472852] Lustre: server umount lustre-MDT0000 complete [ 1193.956154] LustreError: 30076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.983607] LustreError: 30076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1196.525624] LustreError: 30055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316276 with bad export cookie 11945835715083794945 [ 1196.539015] LustreError: 30055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1196.546529] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.373834] Lustre: server umount lustre-MDT0001 complete [ 1210.366168] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1216.078555] Key type lgssc unregistered [ 1216.641875] LNet: 33813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.662055] LNetError: 33813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.689154] LNet: Removed LNI 192.168.201.108@tcp [ 1218.266091] Key type .llcrypt unregistered [ 1218.268101] Key type ._llcrypt unregistered [ 1242.402572] Key type ._llcrypt registered [ 1242.406203] Key type .llcrypt registered [ 1242.610814] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1244.503657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.591337] alg: No test for adler32 (adler32-zlib) [ 1245.738985] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 1245.906796] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1247.567169] Key type lgssc registered [ 1248.963727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.926914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.956366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.402543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.740355] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1269.730682] LustreError: 35084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.753984] LustreError: 35084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1274.857625] LustreError: 35085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.438199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.976454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1282.209087] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1294.355814] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.008941] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1314.725995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1315.181550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.751182] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1329.609396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1340.118254] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1355.241510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.256919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1355.263844] Lustre: Skipped 1 previous similar message [ 1360.360223] LustreError: 36908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.396697] LustreError: 36908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1360.804786] Lustre: server umount lustre-OST0000 complete [ 1369.057952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1369.067672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.083729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1369.092448] Lustre: Skipped 1 previous similar message [ 1370.592216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.594192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1372.798138] Lustre: server umount lustre-MDT0000 complete [ 1375.712490] LustreError: 36210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.728757] LustreError: 36210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1377.713445] LustreError: 35065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316457 with bad export cookie 6719824584788658732 [ 1377.722325] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.191303] Lustre: server umount lustre-MDT0001 complete [ 1397.896288] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1401.759958] Key type lgssc unregistered [ 1402.224634] LNet: 38667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.236892] LNetError: 38667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.271328] LNet: Removed LNI 192.168.201.108@tcp [ 1403.542361] Key type .llcrypt unregistered [ 1403.543926] Key type ._llcrypt unregistered [ 1439.137284] Key type ._llcrypt registered [ 1439.139194] Key type .llcrypt registered [ 1439.297024] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1441.416826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1441.479106] alg: No test for adler32 (adler32-zlib) [ 1442.687801] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 1443.040941] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1444.767238] Key type lgssc registered [ 1446.879043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.957545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.999627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.648217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.754199] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1469.921815] LustreError: 40088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1469.957668] LustreError: 40088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1475.069735] LustreError: 40089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.831664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.596610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.604188] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1493.927767] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1505.415476] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.847811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.300644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.176159] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1529.621846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1534.669780] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1542.828298] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1544.160377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.179167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.202144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.193322] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.214742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1545.218507] Lustre: Skipped 1 previous similar message [ 1547.960249] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1549.626183] Lustre: server umount lustre-MDT0000 complete [ 1550.304518] LustreError: 40817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1550.326232] LustreError: 40817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1553.057210] LustreError: 40083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.624982] LustreError: 40068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316635 with bad export cookie 10962263331591997145 [ 1555.638020] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.655140] LustreError: 40068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1558.189995] LustreError: 40083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.235566] LustreError: 40083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1558.261144] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 1560.551355] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.576355] Lustre: Skipped 1 previous similar message [ 1562.455417] Lustre: server umount lustre-MDT0001 complete [ 1581.981720] Lustre: server umount lustre-OST0000 complete [ 1604.202625] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1608.210571] Key type lgssc unregistered [ 1608.607662] LNet: 43955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.612417] LNetError: 43955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.630241] LNet: Removed LNI 192.168.201.108@tcp [ 1609.711911] Key type .llcrypt unregistered [ 1609.713320] Key type ._llcrypt unregistered [ 1640.107498] Key type ._llcrypt registered [ 1640.109554] Key type .llcrypt registered [ 1640.187803] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1641.624215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1641.653445] alg: No test for adler32 (adler32-zlib) [ 1642.871428] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 1643.236661] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1645.039195] Key type lgssc registered [ 1646.748080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1657.764804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1657.813466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.537423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1664.707620] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1670.118079] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.158777] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1674.539380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.023922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1681.295402] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1689.910384] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1696.755260] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1705.820628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1706.189232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.923971] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1724.235124] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1729.791386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1735.136927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.142083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.150466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1735.157841] Lustre: Skipped 1 previous similar message [ 1735.167371] Lustre: Skipped 1 previous similar message [ 1740.278198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.283223] Lustre: Skipped 2 previous similar messages [ 1741.524144] Lustre: server umount lustre-OST0000 complete [ 1762.349600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.370346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.379849] Lustre: Skipped 1 previous similar message [ 1765.473157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.476535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.539749] Lustre: Skipped 1 previous similar message [ 1769.253179] Lustre: server umount lustre-MDT0000 complete [ 1770.981756] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.002363] LustreError: 45381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1774.489718] LustreError: 45364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785316854 with bad export cookie 15384672271285085199 [ 1774.499044] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.507295] LustreError: 45364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1781.914537] Lustre: server umount lustre-MDT0001 complete [ 1795.488080] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1799.347346] Key type lgssc unregistered [ 1799.854221] LNet: 49096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.860382] LNetError: 49096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.902470] LNet: Removed LNI 192.168.201.108@tcp [ 1801.270264] Key type .llcrypt unregistered [ 1801.276980] Key type ._llcrypt unregistered [ 1836.801785] Key type ._llcrypt registered [ 1836.804517] Key type .llcrypt registered [ 1837.042358] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1839.328815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.712641] alg: No test for adler32 (adler32-zlib) [ 1841.119450] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 1841.673518] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1843.481986] Key type lgssc registered [ 1845.470190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.948447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.966018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.566943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.871522] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1866.728073] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1866.758982] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1871.841355] LustreError: 50516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.355575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.008827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1878.175786] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1887.200300] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1894.221634] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1901.679626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.174385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.923328] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1917.980795] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1926.776311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1934.059814] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1938.626036] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1951.369646] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-29 5h23m47s [ 1953.233375] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1957.857475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.865330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.870932] Lustre: Skipped 2 previous similar messages [ 1957.892585] Lustre: Skipped 2 previous similar messages [ 1958.945978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.956901] Lustre: Skipped 2 previous similar messages [ 1960.666595] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1960.675893] Lustre: Skipped 1 previous similar message [ 1962.466193] Lustre: server umount lustre-MDT0000 complete [ 1965.043359] LustreError: 50516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.576949] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1971.178782] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.205697] LustreError: 50515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1977.174054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.324423] LustreError: 50511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.346414] LustreError: 50511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1977.389167] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.895123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.986462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1983.881808] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1983.972317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1987.259084] Lustre: DEBUG MARKER: Wait for df (36851) ... [ 1989.134802] Lustre: DEBUG MARKER: done [ 1994.730439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.739462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.764703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1998.307536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1998.322776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.331634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1999.341025] Lustre: Skipped 2 previous similar messages [ 2001.101158] Lustre: server umount lustre-OST0000 complete [ 2009.568481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.601160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.609368] Lustre: Skipped 1 previous similar message [ 2012.270327] Lustre: server umount lustre-MDT0000 complete [ 2016.737219] LustreError: 51243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.752385] LustreError: 51243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2017.490583] LustreError: 52359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317097 with bad export cookie 17557363687976518955 [ 2017.508774] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.512168] LustreError: 52359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2018.405083] Lustre: server umount lustre-MDT0001 complete [ 2034.468199] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2038.900296] Key type lgssc unregistered [ 2039.484466] LNet: 55473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.489760] LNetError: 55473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.540730] LNet: Removed LNI 192.168.201.108@tcp [ 2041.107639] Key type .llcrypt unregistered [ 2041.113412] Key type ._llcrypt unregistered [ 2094.296280] Key type ._llcrypt registered [ 2094.298019] Key type .llcrypt registered [ 2094.394477] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2095.931808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.960866] alg: No test for adler32 (adler32-zlib) [ 2097.090104] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 2097.345883] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2099.055582] Key type lgssc registered [ 2099.976409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2109.466771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2109.487648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.867453] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2110.908600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2111.096284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.388242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2128.656501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2128.773564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2128.827808] Lustre: 57683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2129.164100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2134.791386] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2144.027453] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.869444] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.981571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.208837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2161.453356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.752115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2172.234345] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2185.288569] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2192.562404] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-29 5h27m50s [ 2194.071547] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2202.146557] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2205.178867] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2205.234129] LustreError: 57688:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2225.385132] Lustre: lustre-MDT0000: Client 59e14a85-172c-4a73-b6aa-8295161fc141 (at 192.168.201.8@tcp) reconnecting [ 2245.255688] LustreError: 57688:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2248.732607] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2252.166598] Lustre: DEBUG MARKER: done [ 2264.041189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2264.045838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2264.056256] LustreError: Skipped 1 previous similar message [ 2264.061485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2264.091793] Lustre: Skipped 1 previous similar message [ 2267.909421] Lustre: server umount lustre-OST0000 complete [ 2274.273676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.278132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.288850] Lustre: Skipped 1 previous similar message [ 2274.303336] Lustre: Skipped 2 previous similar messages [ 2275.369213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.379482] Lustre: Skipped 1 previous similar message [ 2279.403936] Lustre: server umount lustre-MDT0000 complete [ 2281.453058] LustreError: 57690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.486367] LustreError: 57690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2284.398284] LustreError: 58426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317364 with bad export cookie 8019169410338871491 [ 2284.415661] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.084353] Lustre: server umount lustre-MDT0001 complete [ 2301.167708] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2305.403083] Key type lgssc unregistered [ 2305.775297] LNet: 62022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.785394] LNetError: 62022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.817918] LNet: Removed LNI 192.168.201.108@tcp [ 2307.170590] Key type .llcrypt unregistered [ 2307.172175] Key type ._llcrypt unregistered [ 2369.321943] Key type ._llcrypt registered [ 2369.331202] Key type .llcrypt registered [ 2369.528788] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2371.516172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.700138] alg: No test for adler32 (adler32-zlib) [ 2372.885873] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 2373.258900] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2375.031204] Key type lgssc registered [ 2376.542332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.295952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.324138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.766245] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2382.797640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2383.048152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.481405] Lustre: Failing over lustre-MDT0000 [ 2384.970659] Lustre: server umount lustre-MDT0000 complete [ 2392.206121] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:31:09 (1785317469) [ 2401.010768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.334098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.759925] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2416.091972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.173952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2416.239420] Lustre: 64825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2416.622753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2423.002723] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2433.728677] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2440.095751] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2448.553867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2448.788533] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2448.988603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.333489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2456.459158] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2469.307589] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.478448] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2487.275992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.299565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.301984] Lustre: Skipped 1 previous similar message [ 2487.318037] Lustre: Skipped 2 previous similar messages [ 2492.389929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.399198] Lustre: Skipped 2 previous similar messages [ 2494.063318] Lustre: server umount lustre-MDT0000 complete [ 2497.506138] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.530984] LustreError: 66607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2498.906721] LustreError: 64819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317578 with bad export cookie 16280741201479323079 [ 2498.921614] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2498.927433] LustreError: 64819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2499.763557] Lustre: server umount lustre-MDT0001 complete [ 2504.146580] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2510.335127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2519.007119] Lustre: 63748:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317582/real 1785317582] req@00000000c308f338 x1872041032594304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2519.039125] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2519.054870] Lustre: Skipped 1 previous similar message [ 2520.032507] Lustre: 63746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317583/real 1785317583] req@0000000018691a63 x1872041032594432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2520.641499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.127340] Lustre: 63747:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317587/real 1785317587] req@00000000796ba3e1 x1872041032594560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2524.194388] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe1f0cfdd1a3dd6b1 [ 2524.205840] Lustre: MGC192.168.201.108@tcp: Connection restored to 0@lo (at 0@lo) [ 2524.498410] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2524.582858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.660882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2527.657220] LustreError: 66676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.669983] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2529.765450] Lustre: 63749:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317593/real 1785317593] req@00000000b70b4a49 x1872041032594944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317609 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2529.803174] Lustre: 63749:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2529.941468] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2533.860770] LustreError: 68599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.896352] LustreError: 68599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2534.882227] Lustre: 63746:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317598/real 1785317598] req@00000000cf510b90 x1872041032595072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317614 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2538.998542] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.031240] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2541.598654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.314757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.696102] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2548.234074] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2557.915479] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2564.533837] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2577.894507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.911709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.931461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.697536] Lustre: server umount lustre-MDT0000 complete [ 2583.521917] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.548576] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2585.252461] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317664 with bad export cookie 16280741201479325361 [ 2585.254410] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2585.264027] LustreError: 64820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2585.935563] Lustre: server umount lustre-MDT0001 complete [ 2594.269563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2604.767433] Lustre: 63747:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785317668/real 1785317668] req@00000000a80e02b5 x1872041032624768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785317684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2604.823974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.845643] Lustre: Skipped 2 previous similar messages [ 2608.345300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.593409] LustreError: 71274:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2610.609891] LustreError: 71274:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ecf08f4d x1872041032625152/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1785317690 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2610.662603] LustreError: 71274:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2611.040068] LustreError: 63745:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a697be0c x1872041032625792/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2611.602093] LustreError: 66676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.726309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.868628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2618.905052] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2633.119726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.864642] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2641.536222] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2653.202691] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2662.555228] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2671.073454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.099382] Lustre: Skipped 2 previous similar messages [ 2671.100397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.123063] Lustre: Skipped 4 previous similar messages [ 2676.198758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.206782] Lustre: Skipped 3 previous similar messages [ 2685.407176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2685.783913] Lustre: server umount lustre-MDT0000 complete [ 2686.441825] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.475128] LustreError: 66675:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 2690.922647] LustreError: 64818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317770 with bad export cookie 16280741201479326740 [ 2690.939712] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.951065] LustreError: 64818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2691.557386] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.734534] Lustre: server umount lustre-MDT0001 complete [ 2700.328534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2705.696897] Lustre: DEBUG MARKER: files compared the same [ 2708.470369] Lustre: server umount lustre-OST0000 complete [ 2733.463029] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2737.332206] Key type lgssc unregistered [ 2737.810488] LNet: 74590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2737.826523] LNetError: 74590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2738.852877] LNet: Removed LNI 192.168.201.108@tcp [ 2740.244465] Key type .llcrypt unregistered [ 2740.252064] Key type ._llcrypt unregistered [ 2774.890482] Key type ._llcrypt registered [ 2774.897428] Key type .llcrypt registered [ 2775.015258] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2776.323141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2776.427039] alg: No test for adler32 (adler32-zlib) [ 2777.628666] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 2777.902632] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2779.535150] Key type lgssc registered [ 2781.113158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2790.336132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.348497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.960350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.045558] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2802.145681] LustreError: 76007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.169067] LustreError: 76007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2807.295107] LustreError: 76008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.579151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.977476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2817.281159] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2828.650501] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.599430] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.364084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.859746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.116212] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug +malloc all [ 2862.323472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2864.985747] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2875.871896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2875.877061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2875.900492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.920206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2877.924533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.931776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.900414] Lustre: server umount lustre-OST0000 complete [ 2886.623478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2886.626231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.646223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.654544] Lustre: Skipped 1 previous similar message [ 2893.029280] Lustre: server umount lustre-MDT0000 complete [ 2893.286312] LustreError: 76008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.310341] LustreError: 76008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2897.925112] LustreError: 75987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785317977 with bad export cookie 8093985889062122605 [ 2897.936980] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.363710] Lustre: server umount lustre-MDT0001 complete [ 2915.035626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2920.248876] Key type lgssc unregistered [ 2920.662451] LNet: 79544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2920.676093] LNetError: 79544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2920.713633] LNet: Removed LNI 192.168.201.108@tcp [ 2922.437578] Key type .llcrypt unregistered [ 2922.449930] Key type ._llcrypt unregistered [ 2956.230119] Key type ._llcrypt registered [ 2956.231619] Key type .llcrypt registered [ 2956.354261] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2957.864505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.006513] alg: No test for adler32 (adler32-zlib) [ 2959.395341] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 2959.727074] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2961.471160] Key type lgssc registered [ 2963.214702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2974.038206] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2974.066732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2989.728329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2994.848364] LustreError: 80961:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 3021.535303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.200389] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3039.680581] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3049.447948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.548066] Lustre: *** cfs_fail_loc=706, val=0*** [ 3049.879354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3049.938413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3055.073139] LustreError: 80987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.096969] LustreError: 80987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3055.697459] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3056.116543] LustreError: 80986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.133296] LustreError: 82049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.247657] LustreError: 80987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.260706] LustreError: 80987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3065.515110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.010452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3073.130718] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3082.424638] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3091.864879] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3096.033368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3096.049514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.072343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.573340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3096.593921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.615958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.974809] Lustre: server umount lustre-OST0000 complete [ 3111.905468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3111.921263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.944601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.956347] Lustre: Skipped 2 previous similar messages [ 3113.842729] Lustre: server umount lustre-MDT0000 complete [ 3117.549484] LustreError: 82606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3117.581475] LustreError: 82606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3120.842530] LustreError: 83828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785318200 with bad export cookie 2345287160132261614 [ 3120.846825] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3120.866558] LustreError: 83828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3122.531492] Lustre: server umount lustre-MDT0001 complete [ 3135.360327] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3138.455542] Key type lgssc unregistered [ 3138.701894] LNet: 84581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3138.707315] LNetError: 84581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3138.725603] LNet: Removed LNI 192.168.201.108@tcp [ 3139.460148] Key type .llcrypt unregistered [ 3139.462121] Key type ._llcrypt unregistered [ 3171.405952] Key type ._llcrypt registered [ 3171.412634] Key type .llcrypt registered [ 3171.687101] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3173.541898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.826893] alg: No test for adler32 (adler32-zlib) [ 3175.167308] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 3175.686430] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3177.599374] Key type lgssc registered [ 3180.246012] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3190.646436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3190.667154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.306291] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3206.613636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.824247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3213.286436] LustreError: 86687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.299143] LustreError: 86687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3214.020855] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3217.378222] LustreError: 86685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3217.387308] LustreError: 86685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3222.504795] LustreError: 86687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.526590] LustreError: 86687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3225.765951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.413241] LustreError: 86685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.452621] LustreError: 86685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3226.479409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.712567] LustreError: 86690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.728908] LustreError: 86690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3236.020378] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3242.982126] LustreError: 86691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.992158] LustreError: 86691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3245.361251] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.369701] Lustre: MGS: Client 53578739-d0e3-4a51-bb59-ba8de443bb24 (at 0@lo) reconnecting [ 3245.390598] LustreError: 86701:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3245.675850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.777130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3251.435263] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3271.137596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.153125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.187780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.660770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.285102] Lustre: server umount lustre-OST0000 complete [ 3281.888614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3281.893941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.914685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.927321] Lustre: Skipped 1 previous similar message [ 3281.956362] Lustre: Skipped 2 previous similar messages [ 3286.686594] Lustre: server umount lustre-MDT0000 complete [ 3287.029650] LustreError: 88233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3294.911789] Lustre: server umount lustre-MDT0001 complete [ 3299.515819] Lustre: server umount MGS complete [ 3314.233477] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:46:31 (1785318391) [ 3351.750423] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 3363.930937] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3416.471701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3421.768776] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3435.458184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.950989] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3436.113978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3436.360210] Lustre: lustre-MDT0001: new disk, initializing [ 3436.545429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3436.604075] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3436.617254] Lustre: 93885:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3436.630915] LustreError: 93885:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3443.100622] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3460.128532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3460.724809] Lustre: lustre-OST0000: new disk, initializing [ 3460.735726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3460.752902] Lustre: 94653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3460.897689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.737311] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3468.904346] Lustre: 94948:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000060eb40d4 x1872041874599808/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785318564 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3481.833357] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3489.248763] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3490.547310] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.572081] Lustre: MGS: Client e5357130-8a62-4479-bf25-16a20a830e3e (at 0@lo) reconnecting [ 3490.649432] Lustre: 93152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3490.685422] LustreError: 93871:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3490.715797] LustreError: 93871:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3490.735927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3490.945447] Lustre: lustre-MDT0000: new disk, initializing [ 3491.270197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.331909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3492.393399] Lustre: 95723:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000f802e8a5 x1872041874607360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785318588 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3497.520232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3497.532020] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3497.662950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3498.012093] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3535.328094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3535.330622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.346375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3538.404235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3538.419769] Lustre: Skipped 1 previous similar message [ 3541.026438] Lustre: server umount lustre-OST0000 complete [ 3548.640529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.644875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3548.648046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3548.654185] Lustre: Skipped 2 previous similar messages [ 3552.815032] Lustre: server umount lustre-MDT0000 complete [ 3553.763217] LustreError: 93861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.782980] LustreError: 93861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3572.191139] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3572.401648] Lustre: server umount lustre-MDT0001 complete [ 3576.497290] Lustre: server umount MGS complete [ 3591.003677] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:51:08 (1785318668) [ 3617.535322] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3620.276326] Key type lgssc unregistered [ 3620.519988] LNet: 98219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3620.527266] LNetError: 98219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3620.547191] LNet: Removed LNI 192.168.201.108@tcp [ 3621.615821] Key type .llcrypt unregistered [ 3621.617233] Key type ._llcrypt unregistered [ 3644.154342] Key type ._llcrypt registered [ 3644.158159] Key type .llcrypt registered [ 3644.301315] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3645.642271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3645.897206] alg: No test for adler32 (adler32-zlib) [ 3647.151965] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 3647.525852] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3649.279164] Key type lgssc registered [ 3651.000294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3661.225259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3661.262779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3662.877321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.254723] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3673.064029] LustreError: 99445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.080228] LustreError: 99445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3678.180531] LustreError: 99444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.915936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.573334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3689.571363] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3699.950662] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3707.638207] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3716.478877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3717.043922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.560370] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3732.534075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3737.430803] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3752.943624] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.944378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.955927] LustreError: Skipped 1 previous similar message [ 3752.960378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3753.002491] Lustre: Skipped 1 previous similar message [ 3753.337553] Lustre: server umount lustre-OST0000 complete [ 3763.171544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.185584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.197697] Lustre: Skipped 1 previous similar message [ 3763.222652] Lustre: Skipped 2 previous similar messages [ 3765.323113] Lustre: server umount lustre-MDT0000 complete [ 3768.293812] LustreError: 99444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.314055] LustreError: 99444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3770.956926] LustreError: 99425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785318850 with bad export cookie 8042924812841244253 [ 3770.971339] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3771.595671] Lustre: server umount lustre-MDT0001 complete [ 3786.151557] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3789.820702] Key type lgssc unregistered [ 3790.082937] LNet: 102995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3790.092048] LNetError: 102995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3790.110324] LNet: Removed LNI 192.168.201.108@tcp [ 3791.396094] Key type .llcrypt unregistered [ 3791.397807] Key type ._llcrypt unregistered [ 3812.194074] Key type ._llcrypt registered [ 3812.198332] Key type .llcrypt registered [ 3812.358129] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3814.073382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3814.159753] alg: No test for adler32 (adler32-zlib) [ 3815.236423] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 3815.499022] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3817.183399] Key type lgssc registered [ 3818.548698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3825.827732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3825.834797] LustreError: 104067:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3825.905428] LustreError: 104079:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3825.914046] LustreError: 104067:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4924 [ 3825.920746] LustreError: 104067:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3825.924899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.929567] LustreError: 104067:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3827.784757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.920682] LustreError: 104117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.942275] LustreError: 104117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3838.044316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.656173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.276072] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3852.736467] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing lsmod [ 3857.147792] LustreError: 105276:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3857.225437] LustreError: 105286:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3857.235407] LustreError: 105276:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4923 [ 3857.246050] LustreError: 105276:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3857.263037] LustreError: 105276:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3857.268286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.064944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.512640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3862.540617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3862.543804] Lustre: Skipped 1 previous similar message [ 3862.571581] Lustre: Skipped 2 previous similar messages [ 3863.591773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.605138] Lustre: Skipped 2 previous similar messages [ 3868.642397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.580949] Lustre: server umount lustre-MDT0000 complete [ 3873.770853] LustreError: 104117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.798709] LustreError: 104117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3875.776216] LustreError: 104097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785318955 with bad export cookie 18195186140518532435 [ 3875.781232] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.803683] LustreError: 104097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3876.018206] LustreError: 104548:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb214b97 x1872042545666816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3877.056711] Lustre: server umount lustre-MDT0001 complete [ 3892.058908] Lustre: server umount lustre-OST0000 complete [ 3904.361623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.821780] LustreError: 106350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.893784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.112481] LustreError: 106351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.730671] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3915.232465] LustreError: 106350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3921.070549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.370312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3927.524866] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3938.092742] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3945.803651] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.152549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.688644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.052547] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3975.164930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3977.909066] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3990.500228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.502824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3990.543322] Lustre: Skipped 1 previous similar message [ 3990.546224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.572419] Lustre: Skipped 3 previous similar messages [ 3995.028989] Lustre: server umount lustre-OST0000 complete [ 4000.737658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.738920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.754722] Lustre: Skipped 2 previous similar messages [ 4000.773163] Lustre: Skipped 1 previous similar message [ 4006.178592] Lustre: server umount lustre-MDT0000 complete [ 4010.272178] LustreError: 106333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785319089 with bad export cookie 18195186140518533128 [ 4010.285971] LustreError: 106333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4010.290477] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4010.769720] Lustre: server umount lustre-MDT0001 complete [ 4025.325794] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4028.244623] Key type lgssc unregistered [ 4028.537551] LNet: 109888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4028.541523] LNetError: 109888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4028.550589] LNet: Removed LNI 192.168.201.108@tcp [ 4029.502204] Key type .llcrypt unregistered [ 4029.514868] Key type ._llcrypt unregistered [ 4065.199665] Key type ._llcrypt registered [ 4065.205075] Key type .llcrypt registered [ 4065.438348] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4067.067770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4067.139265] alg: No test for adler32 (adler32-zlib) [ 4069.066411] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 4069.528818] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4071.287856] Key type lgssc registered [ 4074.210026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4088.631808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4088.677358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.104738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.494083] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4099.556127] LustreError: 111310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4099.588041] LustreError: 111310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4104.675784] LustreError: 111309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.246957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.761908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4111.816292] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4121.460758] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4128.795166] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4139.284780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4139.895414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.562385] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4159.362241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4164.591988] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4171.935506] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4179.152294] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4189.152544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4189.159608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.199297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.176085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4190.180136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.202595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.230952] Lustre: Skipped 1 previous similar message [ 4193.363829] Lustre: server umount lustre-OST0000 complete [ 4200.416730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.443422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.286887] Lustre: server umount lustre-MDT0000 complete [ 4205.544185] LustreError: 111310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.563368] LustreError: 111310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4210.509481] LustreError: 111290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785319290 with bad export cookie 11119653966599649037 [ 4210.522383] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.633481] Lustre: server umount lustre-MDT0001 complete [ 4227.991380] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4232.102431] Key type lgssc unregistered [ 4232.486608] LNet: 114963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4232.499245] LNetError: 114963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4233.519183] LNet: Removed LNI 192.168.201.108@tcp [ 4234.797905] Key type .llcrypt unregistered [ 4234.803620] Key type ._llcrypt unregistered [ 4261.645613] Key type ._llcrypt registered [ 4261.648426] Key type .llcrypt registered [ 4261.814800] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4265.699840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4265.747522] alg: No test for adler32 (adler32-zlib) [ 4267.123807] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 4267.402328] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4269.087303] Key type lgssc registered [ 4270.916881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4285.398671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4285.415171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.933423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.576840] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4297.184541] LustreError: 116189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.217524] LustreError: 116189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4302.316051] LustreError: 116190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4305.930172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.465930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.307893] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4321.804925] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4331.174890] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.145460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.611370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.107819] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4357.997277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 4359.272549] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.460575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.467847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.488153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.478508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.480434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.485670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.599938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4378.612375] Lustre: Skipped 1 previous similar message [ 4383.721279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.733841] Lustre: Skipped 1 previous similar message [ 4386.271164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4386.706237] Lustre: server umount lustre-OST0000 complete [ 4393.444369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4393.463752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.521980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.073056] LustreError: 116190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.110857] LustreError: 116190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4399.488231] Lustre: server umount lustre-MDT0000 complete [ 4404.136216] LustreError: 116170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785319483 with bad export cookie 4898093064405647848 [ 4404.142076] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.147749] LustreError: 116170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4404.584776] Lustre: server umount lustre-MDT0001 complete [ 4419.717250] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4424.062028] Key type lgssc unregistered [ 4424.681531] LNet: 119742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4424.691872] LNetError: 119742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4425.702284] LNet: Removed LNI 192.168.201.108@tcp [ 4427.166869] Key type .llcrypt unregistered [ 4427.168255] Key type ._llcrypt unregistered [ 4459.359540] Key type ._llcrypt registered [ 4459.363278] Key type .llcrypt registered [ 4459.542732] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4461.197698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4461.351652] alg: No test for adler32 (adler32-zlib) [ 4462.745270] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 4463.276827] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4465.031155] Key type lgssc registered [ 4466.061659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.247737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.284210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.172609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4485.186269] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4489.704395] LustreError: 121162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.726175] LustreError: 121162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4494.840913] LustreError: 121163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4497.053156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.347706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4501.922753] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4511.480621] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.778709] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4527.813084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4528.511473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4536.991818] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4545.941801] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4550.013306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4558.726278] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4563.475180] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4563.487620] Lustre: Skipped 2 previous similar messages [ 4566.673835] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4566.714964] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4568.946871] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4568.949600] Lustre: Skipped 2 previous similar messages [ 4575.292051] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4575.306289] Lustre: Skipped 1 previous similar message [ 4581.793877] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.8@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4581.804753] Lustre: Skipped 2 previous similar messages [ 4581.996352] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.8@tcp 192.168.0.[2,10]@tcp [ 4595.058478] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.8/24@tcp in log lustre-MDT0000 [ 4595.073803] Lustre: Skipped 2 previous similar messages [ 4604.548846] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.8/24@tcp [ 4604.562787] Lustre: Skipped 1 previous similar message [ 4611.554370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4611.563889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.573258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.607900] Lustre: Skipped 1 previous similar message [ 4616.674363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.676584] Lustre: Skipped 1 previous similar message [ 4617.364803] Lustre: server umount lustre-OST0000 complete [ 4625.386081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.401713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.423076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.425105] Lustre: Skipped 1 previous similar message [ 4626.916852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.032460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4632.049200] Lustre: Skipped 3 previous similar messages [ 4637.160654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4637.169526] Lustre: Skipped 1 previous similar message [ 4638.175789] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4639.137471] Lustre: server umount lustre-MDT0000 complete [ 4642.272784] LustreError: 121162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.301480] LustreError: 121162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4644.033569] LustreError: 124757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785319723 with bad export cookie 1076224683874745918 [ 4644.047246] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4644.732480] Lustre: server umount lustre-MDT0001 complete [ 4657.860125] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4661.604363] Key type lgssc unregistered [ 4661.968261] LNet: 126065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4661.980658] LNetError: 126065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4661.994319] LNet: Removed LNI 192.168.201.108@tcp [ 4663.217309] Key type .llcrypt unregistered [ 4663.221109] Key type ._llcrypt unregistered [ 4698.644616] Key type ._llcrypt registered [ 4698.646805] Key type .llcrypt registered [ 4698.729581] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4699.681737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4699.942698] alg: No test for adler32 (adler32-zlib) [ 4700.919567] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 4701.321461] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4703.103175] Key type lgssc registered [ 4704.383283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4713.216875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4713.240594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.842247] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4714.882847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4714.913149] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.8/24@tcp [ 4720.792800] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4725.222832] LustreError: 127649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.243924] LustreError: 127649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4730.348074] LustreError: 127650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4730.453035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.841238] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4730.881577] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.8/24@tcp [ 4730.958582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4735.856900] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4746.189359] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4754.177274] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4766.151251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4766.855866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4777.800237] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4785.268798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4786.485976] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4800.993143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.997619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4801.014922] Lustre: Skipped 1 previous similar message [ 4801.028822] Lustre: Skipped 1 previous similar message [ 4804.693884] Lustre: server umount lustre-OST0000 complete [ 4811.232197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4811.246731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.254460] Lustre: Skipped 1 previous similar message [ 4815.921368] Lustre: server umount lustre-MDT0000 complete [ 4816.352442] LustreError: 127645:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.366558] LustreError: 127645:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4820.450124] LustreError: 127649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.481777] LustreError: 127649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4820.661258] LustreError: 127630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785319900 with bad export cookie 11845920738339896285 [ 4820.681977] LustreError: 127630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4820.685784] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.053549] Lustre: server umount lustre-MDT0001 complete [ 4835.800635] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4839.629217] Key type lgssc unregistered [ 4839.921377] LNet: 131241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4839.930099] LNetError: 131241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4839.952587] LNet: Removed LNI 192.168.201.108@tcp [ 4840.810773] Key type .llcrypt unregistered [ 4840.816133] Key type ._llcrypt unregistered [ 4872.482813] Key type ._llcrypt registered [ 4872.486850] Key type .llcrypt registered [ 4872.633896] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4891.290518] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4893.660379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4893.747693] alg: No test for adler32 (adler32-zlib) [ 4894.973557] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 4895.312717] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4897.031160] Key type lgssc registered [ 4898.541606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4952.150955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4952.171950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4953.840416] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4953.927783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4954.182878] Lustre: lustre-MDT0000: new disk, initializing [ 4954.346355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.390425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4960.477165] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4975.828501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.030049] Lustre: 134646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4976.106113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4976.116971] Lustre: Skipped 1 previous similar message [ 4976.299924] Lustre: lustre-MDT0001: new disk, initializing [ 4976.488938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4976.571893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4976.596700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4984.175890] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4994.268826] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5001.593196] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5008.799778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5009.104504] Lustre: lustre-OST0000: new disk, initializing [ 5009.109471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5009.116362] Lustre: 136618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5009.239663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5015.082656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5015.087608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5015.231858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5016.432430] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5026.509677] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5031.667508] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5032.036625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5038.432250] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5038.660672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5045.734274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5045.756868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.805389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.966558] Lustre: server umount lustre-OST0000 complete [ 5056.482049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.509471] Lustre: Skipped 1 previous similar message [ 5056.515734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5056.518316] Lustre: Skipped 2 previous similar messages [ 5060.272319] Lustre: server umount lustre-MDT0000 complete [ 5064.985444] LustreError: 134638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785320144 with bad export cookie 3008789866665417101 [ 5064.993496] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.003944] LustreError: 134638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5065.049208] LustreError: 132865:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 5065.538128] Lustre: server umount lustre-MDT0001 complete [ 5074.989299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.386462] LustreError: 138833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5075.400562] LustreError: 138833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5075.482239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.547310] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.038215] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5085.666870] LustreError: 138833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.789258] LustreError: 138834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5092.706076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.298304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5101.588323] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5113.206559] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.196929] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5129.344389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.825278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.677156] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5148.019214] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5154.160685] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5154.450841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5160.152807] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5160.498340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5172.079277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5172.370910] Lustre: lustre-OST0001: new disk, initializing [ 5172.385373] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5172.400797] Lustre: 142208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5172.558105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5179.581128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5179.603682] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5179.785898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5181.181176] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5192.867872] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5193.145762] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5199.289948] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5199.661401] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5206.083831] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff933f95b8d000.ost_server_uuid 50 [ 5208.349813] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff933f95b8d000.ost_server_uuid in IDLE state after 0 sec [ 5220.838781] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5220.847175] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.853125] LustreError: Skipped 1 previous similar message [ 5220.864413] Lustre: Skipped 1 previous similar message [ 5220.865807] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5225.508476] Lustre: server umount lustre-OST0001 complete [ 5225.978776] LustreError: 140657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.007432] LustreError: 140657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5231.085080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.090436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5231.105430] Lustre: Skipped 2 previous similar messages [ 5231.129684] Lustre: Skipped 2 previous similar messages [ 5236.195749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5236.197422] LustreError: 140655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.212690] Lustre: Skipped 1 previous similar message [ 5236.251938] LustreError: 140655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5237.273757] Lustre: server umount lustre-OST0000 complete [ 5246.434059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.437954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.452156] Lustre: Skipped 1 previous similar message [ 5246.471078] Lustre: Skipped 1 previous similar message [ 5251.015535] Lustre: server umount lustre-MDT0000 complete [ 5256.422657] LustreError: 138815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785320336 with bad export cookie 3008789866665417983 [ 5256.432448] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5257.623182] Lustre: server umount lustre-MDT0001 complete [ 5283.768509] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5287.401661] Key type lgssc unregistered [ 5287.797119] LNet: 145214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.812531] LNetError: 145214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.847585] LNet: Removed LNI 192.168.201.108@tcp [ 5289.131387] Key type .llcrypt unregistered [ 5289.133374] Key type ._llcrypt unregistered [ 5346.554440] Key type ._llcrypt registered [ 5346.560860] Key type .llcrypt registered [ 5346.675281] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 5369.541846] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5371.990658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5372.057272] alg: No test for adler32 (adler32-zlib) [ 5373.824584] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 5374.522027] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5376.295244] Key type lgssc registered [ 5377.179371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5425.953129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5426.032081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.647602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5427.783899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5428.009591] Lustre: lustre-MDT0000: new disk, initializing [ 5428.128294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.164388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5434.850830] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5449.470553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5449.669187] Lustre: 149378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5449.810612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5449.816778] Lustre: Skipped 1 previous similar message [ 5449.961608] Lustre: lustre-MDT0001: new disk, initializing [ 5450.105641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5450.158497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5450.175956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5456.880512] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5468.906665] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5477.300377] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5486.132811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5486.610967] Lustre: lustre-OST0000: new disk, initializing [ 5486.613128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5486.633425] Lustre: 151348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5486.776593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.833668] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5494.925895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5494.943605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5495.149379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5505.721590] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5510.966794] Lustre: Failing over lustre-OST0000 [ 5511.489891] Lustre: server umount lustre-OST0000 complete [ 5528.035338] Lustre: 147601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785320591/real 1785320591] req@0000000068ea658c x1872044180410496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785320607 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5528.061170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.627538] Lustre: 147602:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785320595/real 1785320595] req@000000009bf4407c x1872044180411520/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785320611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5532.127546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.398820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5533.864024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5533.885953] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5537.248729] Lustre: 147599:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785320600/real 1785320600] req@000000008028fd8e x1872044180412800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785320616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5537.273829] Lustre: 147599:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5537.951803] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5541.195477] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5541.984085] Lustre: 147601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785320605/real 1785320605] req@0000000040b9e617 x1872044180414080/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785320621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5542.161159] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5542.161537] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 5546.532490] Lustre: Failing over lustre-MDT0000 [ 5546.911849] Lustre: server umount lustre-MDT0000 complete [ 5547.493021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5547.500044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5552.623390] LustreError: 149390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.652423] LustreError: 149390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5553.306363] LustreError: 149384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.734925] LustreError: 150483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5557.764813] LustreError: 150483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5562.858675] LustreError: 149389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5562.880726] LustreError: 149389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5563.874251] Lustre: 147601:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785320627/real 1785320627] req@00000000c8ab16cd x1872044180424832/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785320643 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5563.897482] Lustre: 147601:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5563.911777] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5567.791235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5568.477700] LustreError: 152270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.503154] LustreError: 152270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5574.128564] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x58f53f635f301651 [ 5574.149449] Lustre: MGC192.168.201.108@tcp: Connection restored to 0@lo (at 0@lo) [ 5574.159790] Lustre: Skipped 1 previous similar message [ 5574.688961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5574.789989] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5579.636509] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5579.762256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5579.766521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5579.973710] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5580.093480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5589.992407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.999817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5590.001686] Lustre: Skipped 3 previous similar messages [ 5595.106693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.115500] Lustre: Skipped 2 previous similar messages [ 5596.485097] Lustre: server umount lustre-OST0000 complete [ 5605.349110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.353336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5605.377833] Lustre: Skipped 1 previous similar message [ 5605.401375] Lustre: Skipped 1 previous similar message [ 5607.711779] Lustre: server umount lustre-MDT0000 complete [ 5610.466503] LustreError: 150483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5610.482117] LustreError: 150483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5612.063342] LustreError: 149369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785320691 with bad export cookie 6410099340663068241 [ 5612.071246] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5612.082311] LustreError: 149369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5612.899697] Lustre: server umount lustre-MDT0001 complete [ 5627.662367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5632.357524] Key type lgssc unregistered [ 5632.910518] LNet: 155008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5632.924246] LNetError: 155008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5632.946132] LNet: Removed LNI 192.168.201.108@tcp [ 5634.881467] Key type .llcrypt unregistered [ 5634.883097] Key type ._llcrypt unregistered [ 5691.408314] Key type ._llcrypt registered [ 5691.410266] Key type .llcrypt registered [ 5691.481500] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5692.873038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5692.904426] alg: No test for adler32 (adler32-zlib) [ 5693.946619] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 5694.204876] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5695.887215] Key type lgssc registered [ 5697.704729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5706.276970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.307966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.792099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.480396] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5717.984379] LustreError: 156534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5718.013768] LustreError: 156534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5719.011476] LustreError: 156535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.133174] LustreError: 156534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.235960] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5729.254733] LustreError: 156535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.077114] LustreError: 156563:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5732.091587] LustreError: 156563:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5732.107472] LustreError: 156563:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 5732.904709] Lustre: server umount lustre-MDT0000 complete [ 5753.543694] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 5768.150696] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5815.975843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.166976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5816.197872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5816.279428] Lustre: lustre-MDT0000: new disk, initializing [ 5816.343628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.355814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5822.120288] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5835.348590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.617922] Lustre: 159886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5835.720537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5835.733962] Lustre: Skipped 1 previous similar message [ 5835.898117] Lustre: lustre-MDT0001: new disk, initializing [ 5836.101110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5836.168632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5836.192072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5842.581424] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5856.599487] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5868.255491] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5876.966544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5877.372119] Lustre: lustre-OST0000: new disk, initializing [ 5877.378533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5877.388542] Lustre: 161854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5877.527916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5885.540960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5885.574189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5885.777748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5887.177976] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5899.248352] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5907.442940] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5909.331684] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 06:29:46 (1785320986) [ 6234.079746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6234.080369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6234.104459] Lustre: Skipped 1 previous similar message [ 6239.201186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6239.209382] Lustre: Skipped 4 previous similar messages [ 6244.322307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.367840] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6247.023358] Lustre: server umount lustre-MDT0000 complete [ 6249.439694] LustreError: 159899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6251.925454] LustreError: 159881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785321331 with bad export cookie 10448242803649650118 [ 6251.935468] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6254.560375] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.577295] Lustre: Skipped 1 previous similar message [ 6254.582334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6254.585625] Lustre: Skipped 2 previous similar messages [ 6261.727798] LustreError: 159894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.728079] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6261.745581] LustreError: 159894:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6261.763524] Lustre: Skipped 1 previous similar message [ 6266.335143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6266.799553] Lustre: server umount lustre-MDT0001 complete [ 6287.327169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6288.021946] Lustre: server umount lustre-OST0000 complete [ 6302.738651] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 6315.231727] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6359.859757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.448184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6360.568248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6360.778503] Lustre: lustre-MDT0000: new disk, initializing [ 6360.924690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6360.957699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6367.316409] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6381.505528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.647815] Lustre: 166534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6381.738400] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6381.745588] Lustre: Skipped 1 previous similar message [ 6381.867833] Lustre: lustre-MDT0001: new disk, initializing [ 6381.985789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6382.044056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6382.056490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6387.316593] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6399.465586] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6407.716679] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6417.414874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6417.791542] Lustre: lustre-OST0000: new disk, initializing [ 6417.794457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6417.798814] Lustre: 168506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6417.867624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.189511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6426.209631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6426.470420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6427.038451] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6439.592152] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6446.249614] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6446.646106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6454.042251] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6454.489731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6457.320845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.323027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.336344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6457.345478] Lustre: Skipped 1 previous similar message [ 6471.135662] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6471.886423] Lustre: server umount lustre-OST0000 complete [ 6477.122367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.145047] Lustre: Skipped 1 previous similar message [ 6477.152224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.154223] Lustre: Skipped 4 previous similar messages [ 6483.933191] Lustre: server umount lustre-MDT0000 complete [ 6487.523170] LustreError: 166541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.544910] LustreError: 166541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6489.041676] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785321568 with bad export cookie 10448242803649808955 [ 6489.042150] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.060896] LustreError: 166529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6489.935655] Lustre: server umount lustre-MDT0001 complete [ 6501.278513] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:39:38 (1785321578) [ 6520.938837] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 6534.149528] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6584.734323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.930966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6584.948262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6585.017519] Lustre: lustre-MDT0000: new disk, initializing [ 6585.089091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.105109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6591.019626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6606.098316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.209712] Lustre: 173269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6606.452551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6612.616781] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6626.631042] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6635.393152] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6645.775535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.330671] Lustre: lustre-OST0000: new disk, initializing [ 6646.334723] Lustre: Skipped 1 previous similar message [ 6646.340922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6646.348323] Lustre: Skipped 2 previous similar messages [ 6646.355472] Lustre: 175252:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6651.990609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6651.995226] Lustre: Skipped 1 previous similar message [ 6651.999256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6652.068243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6656.354662] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6668.448479] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.260287] Lustre: server umount lustre-OST0000 complete [ 6688.224493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6688.230018] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.246701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6688.246765] Lustre: Skipped 2 previous similar messages [ 6688.300095] Lustre: Skipped 2 previous similar messages [ 6697.998422] Lustre: server umount lustre-MDT0000 complete [ 6698.409293] Lustre: 155845:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785321762/real 1785321762] req@00000000d4e6b760 x1872044515683456/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785321778 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6698.474118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.521883] LustreError: 173277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6698.564135] LustreError: 173277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6703.629297] Lustre: 155848:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785321767/real 1785321767] req@0000000066d48591 x1872044515686144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785321783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6705.781926] LustreError: 173263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785321785 with bad export cookie 10448242803649809837 [ 6705.798595] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6705.815651] LustreError: 173263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6706.676943] Lustre: server umount lustre-MDT0001 complete [ 6720.566777] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:43:18 (1785321798) [ 6742.656942] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 6763.528786] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6815.281683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.795809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6815.847488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6816.021654] Lustre: lustre-MDT0000: new disk, initializing [ 6816.165482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.168191] Lustre: Skipped 2 previous similar messages [ 6816.207831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6821.828797] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6835.550383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.607324] Lustre: 179603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6835.832500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6842.177178] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6854.950753] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6862.461598] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6871.839269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.343530] Lustre: 181581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6878.794879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6878.907358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6880.301172] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6891.234676] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6904.833659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6904.861148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6904.868705] Lustre: Skipped 1 previous similar message [ 6904.869470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6904.869479] Lustre: Skipped 3 previous similar messages [ 6904.932409] LustreError: Skipped 1 previous similar message [ 6907.365632] Lustre: server umount lustre-OST0000 complete [ 6919.207795] Lustre: server umount lustre-MDT0000 complete [ 6920.171288] LustreError: 179611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6920.197131] LustreError: 179611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6923.734827] LustreError: 179596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322003 with bad export cookie 10448242803649810922 [ 6923.756079] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6924.386172] Lustre: server umount lustre-MDT0001 complete [ 6938.291907] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6941.931232] Key type lgssc unregistered [ 6942.248213] LNet: 183442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.254203] LNetError: 183442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6942.275833] LNet: Removed LNI 192.168.201.108@tcp [ 6943.600888] Key type .llcrypt unregistered [ 6943.605720] Key type ._llcrypt unregistered [ 6971.153215] Key type ._llcrypt registered [ 6971.158265] Key type .llcrypt registered [ 6971.308576] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6972.582430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6972.760057] alg: No test for adler32 (adler32-zlib) [ 6974.007770] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 6974.314223] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6975.994323] Key type lgssc registered [ 6977.247382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6985.692620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6985.710554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6987.125344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6992.928959] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6996.513232] LustreError: 184866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.525220] LustreError: 184866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7001.576914] LustreError: 184867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7003.877506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.561119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7011.746791] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7020.855983] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7027.155893] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7036.205267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7036.845547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7044.490315] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7056.092653] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.355312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7082.977777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7082.986500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.993835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7083.024849] Lustre: Skipped 1 previous similar message [ 7088.025840] Lustre: server umount lustre-OST0000 complete [ 7093.221879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.243839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.253472] Lustre: Skipped 2 previous similar messages [ 7098.336102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.347718] Lustre: Skipped 1 previous similar message [ 7099.683935] Lustre: server umount lustre-MDT0000 complete [ 7102.433616] LustreError: 184866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.444590] LustreError: 184866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7103.939367] LustreError: 184846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322183 with bad export cookie 9042679746674950561 [ 7103.955563] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.956829] LustreError: 184846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7104.439923] Lustre: server umount lustre-MDT0001 complete [ 7117.737574] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7120.532519] Key type lgssc unregistered [ 7120.814684] LNet: 188425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7120.821155] LNetError: 188425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7120.836666] LNet: Removed LNI 192.168.201.108@tcp [ 7121.780842] Key type .llcrypt unregistered [ 7121.782284] Key type ._llcrypt unregistered [ 7151.158887] Key type ._llcrypt registered [ 7151.164916] Key type .llcrypt registered [ 7151.324053] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7153.490029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7153.545761] alg: No test for adler32 (adler32-zlib) [ 7154.902652] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 7155.303894] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7157.063169] Key type lgssc registered [ 7158.496658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7168.183981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7168.224572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.850671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7175.502186] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7180.266886] LustreError: 189848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.296028] LustreError: 189848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7185.395152] LustreError: 189849:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.848054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.364771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7192.529511] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7201.790491] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.704746] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.495953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.078434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.445676] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7232.553835] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7239.669924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7249.897532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.909855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7249.922239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7254.241292] Lustre: lustre-OST0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 7254.247649] Lustre: Skipped 1 previous similar message [ 7255.740995] Lustre: server umount lustre-OST0000 complete [ 7264.386507] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff933f950fb800.ost_server_uuid 50 [ 7266.344526] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7283.682453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7283.689151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.711245] Lustre: Skipped 1 previous similar message [ 7283.722188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.729626] Lustre: Skipped 2 previous similar messages [ 7284.704173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.337509] Lustre: server umount lustre-MDT0000 complete [ 7289.826724] LustreError: 189843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.845626] LustreError: 189843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7292.545130] LustreError: 189829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322372 with bad export cookie 16251294144386288819 [ 7292.569447] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.581730] LustreError: 189829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7293.291225] Lustre: server umount lustre-MDT0001 complete [ 7303.205233] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:53:00 (1785322380) [ 7310.160946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.692533] LustreError: 193789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.768666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.263172] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7315.936591] LustreError: 193790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.880919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7325.277839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7330.028537] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7338.616822] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7344.953944] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7352.935675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7353.374459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7355.437129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7361.016492] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7369.907474] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7377.088391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.395389] Lustre: lustre-OST0001: new disk, initializing [ 7377.401484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7377.412781] Lustre: 196630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7377.539880] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7385.079135] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7385.724100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7385.734948] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7385.884942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7396.600342] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.453333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.465593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7401.473192] Lustre: Skipped 1 previous similar message [ 7401.473271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7401.473287] Lustre: Skipped 2 previous similar messages [ 7406.570126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7406.586112] Lustre: Skipped 3 previous similar messages [ 7406.954744] Lustre: server umount lustre-OST0000 complete [ 7407.786488] LustreError: 195612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.803875] LustreError: 195612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7414.783906] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7414.991673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7420.623210] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7420.862124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7426.205405] LustreError: 195611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.225461] LustreError: 195611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7437.282960] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7437.286573] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7437.300156] Lustre: Skipped 1 previous similar message [ 7439.468222] Lustre: server umount lustre-OST0001 complete [ 7447.522067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.537827] Lustre: Skipped 1 previous similar message [ 7449.950636] Lustre: server umount lustre-MDT0000 complete [ 7453.700880] LustreError: 193769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322533 with bad export cookie 16251294144386289897 [ 7453.710367] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7453.925569] Lustre: server umount lustre-MDT0001 complete [ 7479.680726] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:55:57 (1785322557) [ 7486.782703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.084942] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7487.103194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7487.322685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.258631] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7501.905581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.980847] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7502.018310] Lustre: 200073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7502.110798] LustreError: 200084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.130116] LustreError: 200084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7506.725371] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7515.751218] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7522.437977] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7529.673626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7529.929470] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7537.072806] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7538.834802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7544.166953] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.950203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7551.057455] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7557.056458] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7560.518156] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7565.537667] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.381665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7569.392348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7569.404639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7569.409554] Lustre: Skipped 3 previous similar messages [ 7574.740132] Lustre: server umount lustre-OST0000 complete [ 7576.052321] LustreError: 203430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.074855] LustreError: 203430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7591.400995] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.408762] LustreError: Skipped 2 previous similar messages [ 7597.736415] Lustre: server umount lustre-OST0001 complete [ 7604.710800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7604.717211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7604.729470] Lustre: Skipped 3 previous similar messages [ 7604.735245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7604.743809] Lustre: Skipped 8 previous similar messages [ 7608.250311] Lustre: server umount lustre-MDT0000 complete [ 7611.904111] LustreError: 203714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322691 with bad export cookie 16251294144386291304 [ 7611.914726] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.916158] LustreError: 203714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7612.622322] Lustre: server umount lustre-MDT0001 complete [ 7635.874422] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:58:33 (1785322713) [ 7651.725232] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 7660.932905] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7693.525953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.838839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7693.875670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7693.978096] Lustre: lustre-MDT0000: new disk, initializing [ 7694.070862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7694.079840] Lustre: Skipped 3 previous similar messages [ 7694.106431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7697.433047] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7706.615924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.729273] Lustre: 208168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7706.795943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7706.803479] Lustre: Skipped 1 previous similar message [ 7706.909892] Lustre: lustre-MDT0001: new disk, initializing [ 7707.050755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7707.059325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7711.338705] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7721.393564] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7726.999100] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7734.829242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.170996] Lustre: lustre-OST0000: new disk, initializing [ 7735.174316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7735.182396] Lustre: 210133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7740.412497] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7743.043948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7743.055133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7743.144029] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7749.958896] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7753.473762] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7753.602504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7757.871746] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7758.100630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7763.939653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7763.940927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.948582] LustreError: Skipped 1 previous similar message [ 7763.950558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7763.950563] Lustre: Skipped 2 previous similar messages [ 7763.994102] Lustre: Skipped 2 previous similar messages [ 7774.175160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7774.302800] Lustre: server umount lustre-OST0000 complete [ 7784.378994] Lustre: server umount lustre-MDT0000 complete [ 7784.440899] LustreError: 208173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.467062] LustreError: 208173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 7787.969641] LustreError: 208158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785322867 with bad export cookie 16251294144386293089 [ 7787.973395] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.980387] LustreError: 208158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7788.602597] Lustre: server umount lustre-MDT0001 complete [ 7797.036286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7801.959413] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7810.306746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.292286] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7823.144870] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7828.627736] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7836.077534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7836.391455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7836.395199] Lustre: Skipped 4 previous similar messages [ 7842.344984] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7850.293351] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.167421] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7856.471383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7861.149324] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7861.408753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7870.015147] Lustre: server umount lustre-OST0000 complete [ 7877.231564] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7881.767679] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7912.076117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.327951] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7927.078188] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7930.066203] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7933.092608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7933.121057] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7933.121853] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7933.150858] Lustre: Skipped 1 previous similar message [ 7938.529323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7938.531764] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7938.541231] Lustre: Skipped 4 previous similar messages [ 7938.544806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7938.551270] LustreError: Skipped 1 previous similar message [ 7938.571168] Lustre: Skipped 11 previous similar messages [ 7941.271095] Lustre: server umount lustre-OST0000 complete [ 7951.980518] Lustre: server umount lustre-MDT0000 complete [ 7956.363824] LustreError: 212345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323036 with bad export cookie 16251294144386293971 [ 7956.366502] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7956.376193] LustreError: 212345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7957.038457] Lustre: server umount lustre-MDT0001 complete [ 7964.783449] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:04:02 (1785323042) [ 7971.500337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.794392] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7985.807578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7990.032404] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7996.716691] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8001.504101] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8007.394373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.236853] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8022.004341] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8028.821053] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8028.975270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8032.608165] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8032.841249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8038.331059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8038.498444] Lustre: lustre-OST0001: new disk, initializing [ 8038.502973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8038.510254] Lustre: 221700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8043.659566] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8044.128844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8044.139977] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8044.223913] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8051.944243] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8055.733888] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8055.895776] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8059.522377] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8059.681087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8064.992353] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.007589] LustreError: Skipped 2 previous similar messages [ 8067.565381] Lustre: server umount lustre-OST0001 complete [ 8070.118523] LustreError: 220625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8070.132720] LustreError: 220625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 8073.303734] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8073.486894] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8077.718092] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8077.954282] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8105.688314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.903304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8105.911775] Lustre: Skipped 5 previous similar messages [ 8111.074494] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8111.090836] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8111.093516] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8111.847883] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8118.325838] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8119.874395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8127.969029] Lustre: server umount lustre-OST0001 complete [ 8150.711875] Lustre: server umount lustre-OST0000 complete [ 8160.383070] Lustre: server umount lustre-MDT0000 complete [ 8163.681246] LustreError: 218311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323243 with bad export cookie 16251294144386294937 [ 8163.688803] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.694965] LustreError: 218311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8170.004228] Lustre: server umount lustre-MDT0001 complete [ 8190.838647] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:07:48 (1785323268) [ 8196.847936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.165456] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8197.215527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8200.929518] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8208.092391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.185273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8208.231928] Lustre: 226944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8213.130967] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8220.324396] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8225.157987] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8230.938933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8231.118774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8236.436157] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8242.561545] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8250.726035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8250.803760] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8256.499329] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8263.287822] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8264.418914] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8267.702392] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8267.838140] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8272.065617] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8272.264242] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8277.225399] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff933f86ec3000.ost_server_uuid 50 [ 8278.634720] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff933f86ec3000.ost_server_uuid in IDLE state after 0 sec [ 8279.743866] Lustre: Permanently deactivating lustre-OST0001 [ 8279.761399] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8282.779906] Lustre: Permanently reactivating lustre-OST0001 [ 8286.692064] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8286.698155] LustreError: Skipped 3 previous similar messages [ 8286.702874] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8286.716246] Lustre: Skipped 11 previous similar messages [ 8286.725283] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8286.727497] Lustre: Skipped 13 previous similar messages [ 8292.647615] Lustre: server umount lustre-OST0001 complete [ 8302.336913] Lustre: server umount lustre-OST0000 complete [ 8312.596920] Lustre: server umount lustre-MDT0000 complete [ 8315.860613] LustreError: 226936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323395 with bad export cookie 16251294144386296225 [ 8315.869626] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8315.875338] LustreError: 226936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8316.481702] Lustre: server umount lustre-MDT0001 complete [ 8336.670372] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:10:14 (1785323414) [ 8343.761709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.143085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8344.200049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8344.208825] Lustre: Skipped 5 previous similar messages [ 8348.699107] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8356.497495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.591780] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8356.618051] Lustre: 233628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8360.675717] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8367.696783] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8372.701762] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8378.252298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8378.455883] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8378.495857] Lustre: Permanently deactivating lustre-OST0000 [ 8378.516364] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8383.957877] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8385.055564] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8389.535129] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8395.300720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8395.392684] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8400.442671] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8402.598878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8407.740797] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8409.602451] Lustre: Permanently reactivating lustre-OST0000 [ 8409.614483] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8409.619280] Lustre: Skipped 2 previous similar messages [ 8418.384024] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 0000000035fd4591, cur 1785323498 deadline 1785323494 last 1785323464 [ 8418.384159] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8418.398960] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8418.409250] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8418.415021] Lustre: Skipped 1 previous similar message [ 8418.451042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8423.280376] Lustre: server umount lustre-OST0001 complete [ 8433.086731] Lustre: server umount lustre-OST0000 complete [ 8443.236587] Lustre: server umount lustre-MDT0000 complete [ 8446.601794] LustreError: 233620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323526 with bad export cookie 16251294144386297863 [ 8446.619200] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8446.623724] LustreError: 233620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8447.170749] Lustre: server umount lustre-MDT0001 complete [ 8456.123569] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8458.648975] Key type lgssc unregistered [ 8458.939515] LNet: 238485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8458.949088] LNetError: 238485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8458.966961] LNet: Removed LNI 192.168.201.108@tcp [ 8459.801905] Key type .llcrypt unregistered [ 8459.805620] Key type ._llcrypt unregistered [ 8477.392164] Key type ._llcrypt registered [ 8477.395082] Key type .llcrypt registered [ 8477.490783] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8478.301937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8478.433633] alg: No test for adler32 (adler32-zlib) [ 8479.472764] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 8479.714455] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8481.367131] Key type lgssc registered [ 8482.269331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8490.697643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8490.721640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.265105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.980466] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8501.736871] LustreError: 239946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.758511] LustreError: 239946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8503.461468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.598785] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8503.628177] Lustre: 239936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8503.638152] Lustre: Permanently deactivating lustre-MDT0001 [ 8503.657627] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8503.972291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8506.516578] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8506.522802] LustreError: 239977:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8506.538112] LustreError: 239977:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8506.551721] LustreError: 239977:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 8508.225813] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8514.431300] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8519.635244] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8525.921318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.385252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.425840] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.127631] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8533.481457] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.553261] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.567680] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8536.572222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8538.321747] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8541.670798] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.690051] LustreError: 241787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8544.182562] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8544.426410] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8549.880240] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8549.896221] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8550.323218] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8556.456249] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8558.982888] Lustre: Permanently reactivating lustre-MDT0001 [ 8559.004486] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8559.011508] Lustre: Skipped 2 previous similar messages [ 8563.900171] Lustre: lustre-MDT0001: Client 4c390b70-260c-4ea3-82d8-73b4268eb909 (at 192.168.201.8@tcp) reconnecting [ 8568.902225] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8570.107180] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8574.805834] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8574.953023] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8578.443438] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8578.657865] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8580.865495] Lustre: Permanently deactivating lustre-MDT0001 [ 8580.889772] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8580.892674] Lustre: Skipped 2 previous similar messages [ 8589.382616] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8595.939246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8595.954728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8595.982591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8597.994644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8597.996763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8597.999791] Lustre: Skipped 1 previous similar message [ 8598.012270] Lustre: Skipped 1 previous similar message [ 8600.863942] Lustre: server umount lustre-MDT0000 complete [ 8603.108769] LustreError: 239947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8603.134071] LustreError: 239947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8603.952476] LustreError: 241806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323683 with bad export cookie 9808531731834971345 [ 8603.958508] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8603.978268] LustreError: 241806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8604.510334] Lustre: server umount lustre-MDT0001 complete [ 8610.134896] Lustre: server umount lustre-OST0000 complete [ 8619.224337] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 07:14:57 (1785323697) [ 8628.407848] Lustre: server umount lustre-OST0001 complete [ 8634.262388] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 8642.074475] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8672.075348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8672.261471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8672.266396] Lustre: Skipped 2 previous similar messages [ 8672.286321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8672.405275] Lustre: lustre-MDT0000: new disk, initializing [ 8672.545126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.569995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8675.877401] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8684.485410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8684.599546] Lustre: 248202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8684.673439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8684.679087] Lustre: Skipped 1 previous similar message [ 8684.768120] Lustre: lustre-MDT0001: new disk, initializing [ 8684.849201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8684.879652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8684.893548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8688.134208] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8695.871174] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8701.167865] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8706.328221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8706.653633] Lustre: lustre-OST0000: new disk, initializing [ 8706.659285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8706.669307] Lustre: 250169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8706.791549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8712.277825] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8716.369710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8716.382384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8716.536866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8720.119810] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8721.852793] LustreError: 248208:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8729.899972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8730.051239] Lustre: lustre-OST0001: new disk, initializing [ 8730.054574] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8730.059221] Lustre: 251285:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8731.944267] LustreError: 248208:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8731.998466] LustreError: 251005:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8736.049242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8738.905724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8738.930758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8739.064196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8742.040230] LustreError: 251005:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8744.660893] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8746.114093] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8748.213970] Lustre: lustre-OST0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 8748.216449] Lustre: Skipped 2 previous similar messages [ 8749.536824] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.537480] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.548197] LustreError: Skipped 1 previous similar message [ 8749.561889] Lustre: Skipped 2 previous similar messages [ 8753.306608] Lustre: lustre-OST0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 8753.314301] Lustre: Skipped 2 previous similar messages [ 8754.115104] Lustre: server umount lustre-OST0001 complete [ 8754.665167] LustreError: 250155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.675908] LustreError: 250155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8773.089148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.094768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.101866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8773.102796] Lustre: Skipped 1 previous similar message [ 8773.109994] Lustre: Skipped 1 previous similar message [ 8776.970493] Lustre: server umount lustre-OST0000 complete [ 8780.258512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.271108] Lustre: Skipped 1 previous similar message [ 8785.377208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8785.386243] Lustre: Skipped 5 previous similar messages [ 8786.264667] Lustre: server umount lustre-MDT0000 complete [ 8789.227787] LustreError: 248195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785323868 with bad export cookie 9808531731834973942 [ 8789.237577] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.242298] LustreError: 248195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8795.550673] Lustre: server umount lustre-MDT0001 complete [ 8805.619940] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8808.235648] Key type lgssc unregistered [ 8808.510387] LNet: 253317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8808.516363] LNetError: 253317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8808.527091] LNet: Removed LNI 192.168.201.108@tcp [ 8809.270428] Key type .llcrypt unregistered [ 8809.271625] Key type ._llcrypt unregistered [ 8843.227429] Key type ._llcrypt registered [ 8843.230027] Key type .llcrypt registered [ 8843.322504] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 8844.374154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8844.438340] alg: No test for adler32 (adler32-zlib) [ 8845.559338] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 8845.843444] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 8847.511166] Key type lgssc registered [ 8848.590956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8855.282662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8855.303092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.568474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8856.588803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8856.747067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8859.844971] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8867.248521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8867.358655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8867.409817] Lustre: 255526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8867.748614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8871.656657] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8877.126516] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8881.617853] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8887.303094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.496489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8887.713546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.829751] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8897.107088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8898.177539] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8948.193316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8948.197797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8948.201147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8948.211491] Lustre: Skipped 1 previous similar message [ 8948.218165] Lustre: Skipped 1 previous similar message [ 8950.517562] Lustre: server umount lustre-OST0000 complete [ 8953.716308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8963.038588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8963.375341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.319438] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 8968.486743] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8968.496351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8968.503257] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8968.513397] Lustre: Skipped 1 previous similar message [ 8972.927485] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9014.753275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.754805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9014.768365] LustreError: Skipped 1 previous similar message [ 9014.770043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.388359] Lustre: server umount lustre-OST0000 complete [ 9024.993787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9024.994720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9025.000465] Lustre: Skipped 2 previous similar messages [ 9025.005132] Lustre: Skipped 2 previous similar messages [ 9028.807471] Lustre: server umount lustre-MDT0000 complete [ 9031.622724] LustreError: 257396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324111 with bad export cookie 13532862389911866413 [ 9031.632256] LustreError: 257396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9031.633239] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9032.140453] Lustre: server umount lustre-MDT0001 complete [ 9038.842912] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9040.950077] Key type lgssc unregistered [ 9041.204783] LNet: 260935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9041.211883] LNetError: 260935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9041.221967] LNet: Removed LNI 192.168.201.108@tcp [ 9042.029441] Key type .llcrypt unregistered [ 9042.030914] Key type ._llcrypt unregistered [ 9069.669409] Key type ._llcrypt registered [ 9069.670628] Key type .llcrypt registered [ 9069.738218] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9070.498102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9070.555976] alg: No test for adler32 (adler32-zlib) [ 9071.583297] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9071.822021] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9073.511214] Key type lgssc registered [ 9074.315753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9079.294526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9079.308815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.621246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9082.962358] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9088.185990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.497897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9091.276799] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9097.118755] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9100.757064] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9105.587193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9105.841326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.872105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 9109.907832] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9114.845568] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9147.877048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9147.879476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.889991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9149.924122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9149.928325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9149.934578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9152.162715] Lustre: server umount lustre-OST0000 complete [ 9155.042117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9155.042660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9155.067639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9155.069369] Lustre: Skipped 1 previous similar message [ 9160.173458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.176072] Lustre: Skipped 1 previous similar message [ 9161.386774] Lustre: server umount lustre-MDT0000 complete [ 9164.201975] LustreError: 262330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324243 with bad export cookie 11180536412192677410 [ 9164.210598] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9164.211181] LustreError: 262330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9164.583257] Lustre: server umount lustre-MDT0001 complete [ 9171.845692] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9173.637018] Key type lgssc unregistered [ 9173.870070] LNet: 266664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9173.874574] LNetError: 266664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9173.889182] LNet: Removed LNI 192.168.201.108@tcp [ 9174.558478] Key type .llcrypt unregistered [ 9174.559900] Key type ._llcrypt unregistered [ 9186.706763] Key type ._llcrypt registered [ 9186.709860] Key type .llcrypt registered [ 9186.810020] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9187.653063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9187.751080] alg: No test for adler32 (adler32-zlib) [ 9188.796176] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9189.064169] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9190.767218] Key type lgssc registered [ 9191.407055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9196.426098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9196.443353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.719514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9200.551489] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9206.311945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9206.607736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9209.493083] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9214.080073] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9217.711485] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9222.461319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9222.779049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.855722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9227.395186] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9233.143731] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9273.317407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9273.325028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.339533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9273.344520] Lustre: Skipped 1 previous similar message [ 9277.928566] Lustre: server umount lustre-OST0000 complete [ 9283.558752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9283.563339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9283.570566] Lustre: Skipped 1 previous similar message [ 9283.577110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9283.588187] Lustre: Skipped 1 previous similar message [ 9288.681576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.688621] Lustre: Skipped 1 previous similar message [ 9293.797496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9293.799159] Lustre: Skipped 1 previous similar message [ 9295.327075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9295.559928] Lustre: server umount lustre-MDT0000 complete [ 9298.351891] LustreError: 267841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324378 with bad export cookie 6109785147154263399 [ 9298.365012] LustreError: 267841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9298.366198] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9298.875733] Lustre: server umount lustre-MDT0001 complete [ 9306.680148] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9308.902279] Key type lgssc unregistered [ 9309.139256] LNet: 271558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9309.148886] LNetError: 271558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9309.165663] LNet: Removed LNI 192.168.201.108@tcp [ 9309.879194] Key type .llcrypt unregistered [ 9309.881095] Key type ._llcrypt unregistered [ 9327.636181] Key type ._llcrypt registered [ 9327.640097] Key type .llcrypt registered [ 9327.734575] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9328.440413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9328.468417] alg: No test for adler32 (adler32-zlib) [ 9329.472092] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9329.714139] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9331.375139] Key type lgssc registered [ 9332.276575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9337.922485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9337.947889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9339.204121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9342.281421] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9348.577629] LustreError: 272954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9348.594321] LustreError: 272954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9349.313623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9349.619486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9352.614425] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9358.221190] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9362.443212] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9367.466076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9367.747990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9372.321440] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9377.863416] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9384.183791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9405.923197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9405.931649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9405.954000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9410.017324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9410.023255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9410.024889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.619743] Lustre: server umount lustre-OST0000 complete [ 9415.136819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9415.140067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9415.154937] Lustre: Skipped 1 previous similar message [ 9415.164919] Lustre: Skipped 1 previous similar message [ 9420.259139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9420.263306] Lustre: Skipped 2 previous similar messages [ 9420.854497] Lustre: server umount lustre-MDT0000 complete [ 9423.937364] LustreError: 273689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324503 with bad export cookie 17615620763441196846 [ 9423.939262] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9423.945106] LustreError: 273689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9424.307655] Lustre: server umount lustre-MDT0001 complete [ 9432.217367] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9434.979890] Key type lgssc unregistered [ 9435.320489] LNet: 277277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9435.327835] LNetError: 277277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9435.347710] LNet: Removed LNI 192.168.201.108@tcp [ 9436.275775] Key type .llcrypt unregistered [ 9436.278286] Key type ._llcrypt unregistered [ 9448.992879] Key type ._llcrypt registered [ 9448.994285] Key type .llcrypt registered [ 9449.083122] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9450.066673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9450.099069] alg: No test for adler32 (adler32-zlib) [ 9451.107417] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9451.286457] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9452.919209] Key type lgssc registered [ 9453.843772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9459.651251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9459.672692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.260653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.585335] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9470.434450] LustreError: 278492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9470.446217] LustreError: 278492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9470.759438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.078982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9474.060723] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9479.356110] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9483.873054] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9488.775996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.082737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9493.372308] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9498.947334] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9505.461529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9517.024708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9517.031755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9517.045542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.132957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9521.135146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9521.142306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9521.166069] Lustre: Skipped 1 previous similar message [ 9522.788181] Lustre: server umount lustre-OST0000 complete [ 9526.249238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.265425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9526.270944] Lustre: Skipped 1 previous similar message [ 9531.369751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9531.379119] Lustre: Skipped 1 previous similar message [ 9532.083485] Lustre: server umount lustre-MDT0000 complete [ 9534.968620] LustreError: 278458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324614 with bad export cookie 12943047855575229197 [ 9534.969350] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.976835] LustreError: 278458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9535.322840] Lustre: server umount lustre-MDT0001 complete [ 9543.530931] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9545.546131] Key type lgssc unregistered [ 9545.778143] LNet: 282213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9545.782406] LNetError: 282213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9545.795897] LNet: Removed LNI 192.168.201.108@tcp [ 9546.433224] Key type .llcrypt unregistered [ 9546.436194] Key type ._llcrypt unregistered [ 9554.344350] Key type ._llcrypt registered [ 9554.345779] Key type .llcrypt registered [ 9554.442876] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9566.550784] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 9576.652372] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9577.570471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9577.653665] alg: No test for adler32 (adler32-zlib) [ 9578.706191] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9578.963298] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9580.655195] Key type lgssc registered [ 9581.305546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9606.107458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.129446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.369768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9607.408308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9607.507289] Lustre: lustre-MDT0000: new disk, initializing [ 9607.585512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.605626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.492347] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9618.268991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.334366] Lustre: 285740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9618.366562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9618.369505] Lustre: Skipped 1 previous similar message [ 9618.419218] Lustre: lustre-MDT0001: new disk, initializing [ 9618.479455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9618.508597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9618.516184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9621.746174] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9628.731986] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.821576] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.769168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9638.020838] Lustre: lustre-OST0000: new disk, initializing [ 9638.026691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9638.034150] Lustre: 287707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9638.100107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.345846] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9644.590045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9644.604224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9644.728191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9649.810927] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9653.201167] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9653.379485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9656.436910] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9656.593072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9660.390593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9660.400689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9660.416617] LustreError: Skipped 1 previous similar message [ 9660.419618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9660.431143] Lustre: Skipped 1 previous similar message [ 9664.422735] Lustre: server umount lustre-OST0000 complete [ 9669.600192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9669.607121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.621880] Lustre: Skipped 1 previous similar message [ 9669.628974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.648238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.659737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.662481] Lustre: Skipped 1 previous similar message [ 9675.744598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9675.748327] Lustre: Skipped 1 previous similar message [ 9680.866618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9680.870420] Lustre: Skipped 1 previous similar message [ 9681.375094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9681.617599] Lustre: server umount lustre-MDT0000 complete [ 9684.463407] LustreError: 285735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324764 with bad export cookie 14162079992532015610 [ 9684.464953] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9684.471967] LustreError: 285735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9685.076788] Lustre: server umount lustre-MDT0001 complete [ 9691.381510] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 07:32:49 (1785324769) [ 9696.449584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9696.898901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9700.148651] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9702.369989] LustreError: 290120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.386482] LustreError: 290120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9706.466223] LustreError: 290119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.777433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.089468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9710.304241] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9715.349847] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9719.517269] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.075336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.386441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9729.102550] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9734.928578] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9753.058833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.059570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.069425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9758.486482] Lustre: server umount lustre-OST0000 complete [ 9763.297027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9763.299382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9763.318129] Lustre: Skipped 1 previous similar message [ 9767.473098] Lustre: server umount lustre-MDT0000 complete [ 9768.415965] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9768.426131] LustreError: 290114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9770.135842] LustreError: 290099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785324849 with bad export cookie 14162079992532016492 [ 9770.141650] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.144010] LustreError: 290099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9770.653958] Lustre: server umount lustre-MDT0001 complete [ 9777.532336] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 9779.756860] Key type lgssc unregistered [ 9779.996688] LNet: 293675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9779.999924] LNetError: 293675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9781.042888] LNet: Removed LNI 192.168.201.108@tcp [ 9781.935862] Key type .llcrypt unregistered [ 9781.938237] Key type ._llcrypt unregistered [ 9801.455236] Key type ._llcrypt registered [ 9801.457684] Key type .llcrypt registered [ 9801.556469] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 9812.755505] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 9813.865843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9813.883746] alg: No test for adler32 (adler32-zlib) [ 9814.896710] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [ 9815.137274] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 9816.807175] Key type lgssc registered [ 9817.967299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9848.678635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9848.703423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.933045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9849.959255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9850.083700] Lustre: lustre-MDT0000: new disk, initializing [ 9850.179983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9850.202668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9853.556583] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9861.539090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.658093] Lustre: 297476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9861.715267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9861.717263] Lustre: Skipped 1 previous similar message [ 9861.800302] Lustre: lustre-MDT0001: new disk, initializing [ 9861.889851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9861.933457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9861.939441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9865.121760] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9872.691468] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9876.723336] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9881.801040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9882.046201] Lustre: lustre-OST2710: new disk, initializing [ 9882.049460] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9882.056552] Lustre: 299443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9882.132937] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9886.500261] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9888.298939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9888.308237] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9888.395077] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9893.154208] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9898.182059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9898.341858] Lustre: lustre-OST03e8: new disk, initializing [ 9898.347690] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9898.357828] Lustre: 300506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9898.446743] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9903.073090] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 9906.224708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9906.234878] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9906.310901] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9908.340214] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9913.287194] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9913.445607] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9916.599809] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9916.741251] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9919.856289] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9919.993612] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9922.772264] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9922.894355] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9927.140569] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.143074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9927.144763] Lustre: Skipped 3 previous similar messages [ 9927.146704] Lustre: Skipped 2 previous similar messages [ 9932.264728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9932.268491] Lustre: Skipped 4 previous similar messages [ 9933.362720] Lustre: server umount lustre-MDT0000 complete [ 9935.836907] LustreError: 298269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325015 with bad export cookie 74574569300294063 [ 9935.840634] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9935.848084] LustreError: 298269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9936.422618] Lustre: server umount lustre-MDT0001 complete [ 9948.544346] Lustre: server umount lustre-OST2710 complete [ 9960.626879] Lustre: server umount lustre-OST03e8 complete [ 9972.308772] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 9977.929466] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10004.438869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10004.675147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10004.715512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10004.828775] Lustre: lustre-MDT0000: new disk, initializing [10004.907258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10004.925241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10008.143399] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10016.071543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.179044] Lustre: 305749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10016.235033] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10016.239764] Lustre: Skipped 1 previous similar message [10016.321253] Lustre: lustre-MDT0001: new disk, initializing [10016.416187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10016.448488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10016.460119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10019.830842] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10028.173283] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10032.535791] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10037.874616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10038.129676] Lustre: lustre-OST0000: new disk, initializing [10038.134692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10038.143936] Lustre: 307715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10038.236024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.478685] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10046.526337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10046.540863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10046.658823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10049.795138] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10052.818736] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10052.939078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10056.092624] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10056.244451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10062.310572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10062.312961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10062.318937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10062.327167] Lustre: Skipped 1 previous similar message [10063.872454] Lustre: server umount lustre-OST0000 complete [10067.430290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.440340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10067.444498] Lustre: Skipped 2 previous similar messages [10072.547363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10072.556404] Lustre: Skipped 1 previous similar message [10072.807540] Lustre: server umount lustre-MDT0000 complete [10075.784153] LustreError: 306542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325155 with bad export cookie 74574569300295743 [10075.785241] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10075.790979] LustreError: 306542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10076.247093] Lustre: server umount lustre-MDT0001 complete [10082.350813] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:39:20 (1785325160) [10083.274620] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10084.383546] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:39:22 (1785325162) [10086.767352] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10107.087460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10107.396447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10107.419110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10107.577952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.607686] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10116.607906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.701393] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10116.746135] Lustre: 311569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10120.222468] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10139.503766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10139.719898] Lustre: Denying initial registration attempt from nid 192.168.201.108@tcp, specified as failover [10139.727111] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10139.734803] LustreError: 313022:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10139.745102] LustreError: 313022:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10139.750133] LustreError: 313022:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10139.826193] Lustre: server umount lustre-OST0000 complete [10139.830249] LustreError: 313022:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [10147.807967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10147.812709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.823487] Lustre: Skipped 1 previous similar message [10147.828418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10159.071298] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10159.353691] Lustre: server umount lustre-MDT0000 complete [10162.168598] LustreError: 311562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325241 with bad export cookie 74574569300296625 [10162.177807] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10162.183696] LustreError: 311562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10162.380833] Lustre: server umount lustre-MDT0001 complete [10169.978748] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10172.039209] Key type lgssc unregistered [10172.257378] LNet: 314142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10172.260195] LNetError: 314142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10172.277970] LNet: Removed LNI 192.168.201.108@tcp [10172.931372] Key type .llcrypt unregistered [10172.935149] Key type ._llcrypt unregistered [10180.167356] Key type ._llcrypt registered [10180.169692] Key type .llcrypt registered [10180.236959] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10181.208642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10181.229063] alg: No test for adler32 (adler32-zlib) [10182.225270] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [10182.429084] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10184.055109] Key type lgssc registered [10185.037739] Lustre: Echo OBD driver; http://www.lustre.org/ [10220.901319] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10227.203832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10227.228487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.492242] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10228.518164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10228.694437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10231.529479] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10237.271510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.383989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10237.422929] Lustre: 316886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10237.678221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10240.448445] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10245.433816] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10249.012578] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10266.292125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10266.491816] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10266.630487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.674922] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10275.865536] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10277.859188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10277.864670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.882117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10278.370783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10278.372248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.396834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10278.399440] Lustre: Skipped 1 previous similar message [10279.621268] Lustre: server umount lustre-OST0000 complete [10283.488646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10283.499340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10283.504642] Lustre: Skipped 1 previous similar message [10288.776598] Lustre: server umount lustre-MDT0000 complete [10291.367836] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325371 with bad export cookie 5673900393300613137 [10291.373054] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10291.378354] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10291.760076] Lustre: server umount lustre-MDT0001 complete [10298.984569] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10300.982347] Key type lgssc unregistered [10301.238112] LNet: 320454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10301.245235] LNetError: 320454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10301.266224] LNet: Removed LNI 192.168.201.108@tcp [10302.006985] Key type .llcrypt unregistered [10302.008323] Key type ._llcrypt unregistered [10318.128975] Key type ._llcrypt registered [10318.130189] Key type .llcrypt registered [10318.226916] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10318.958097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10319.024128] alg: No test for adler32 (adler32-zlib) [10320.073285] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [10320.312837] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10321.967160] Key type lgssc registered [10322.953298] Lustre: Echo OBD driver; http://www.lustre.org/ [10328.644523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10328.653256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.133369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10333.168539] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10339.293565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.545033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10342.469214] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10347.602948] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10351.095759] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10357.748119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10358.070511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10362.208602] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10367.107712] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10383.022287] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [10384.669890] Lustre: server umount lustre-MDT0000 complete [10385.889626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.895343] LustreError: 321850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.905884] Lustre: Skipped 1 previous similar message [10385.930619] LustreError: 321850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10387.078755] LustreError: 321829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325466 with bad export cookie 10918528195565884267 [10387.089811] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10388.121385] LustreError: 321844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10388.152319] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [10389.771657] Lustre: server umount lustre-MDT0001 complete [10394.025533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10399.853809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.370581] Lustre: 321161:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325470/real 1785325470] req@0000000068e37cba x1872049365610240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785325486 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10406.386346] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.396303] Lustre: Skipped 1 previous similar message [10407.455174] Lustre: 321161:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325471/real 1785325471] req@000000004b356a08 x1872049365610368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785325487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10411.487192] Lustre: 321163:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325475/real 1785325475] req@00000000cba871a1 x1872049365610496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785325491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10411.554496] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9786678b86caaf4f [10411.566200] Lustre: MGC192.168.201.108@tcp: Connection restored to 0@lo (at 0@lo) [10411.781026] LustreError: 323680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10411.843439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10411.899921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10415.090019] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10416.929299] LustreError: 325461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.947615] LustreError: 325461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10418.015077] Lustre: 321160:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325481/real 1785325481] req@000000007cd1ac24 x1872049365610880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785325497 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10418.033788] Lustre: 321160:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10421.023240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.414167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.303727] Lustre: 321161:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325485/real 1785325485] req@00000000ca4e073d x1872049365611136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785325501 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10424.413679] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10425.520750] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10430.205284] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10431.289467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10434.896190] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10435.872910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10441.189838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10441.196244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10441.203035] Lustre: Skipped 1 previous similar message [10441.217577] Lustre: Skipped 1 previous similar message [10445.293577] Lustre: server umount lustre-OST0000 complete [10451.427645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10451.434635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10451.440046] Lustre: Skipped 1 previous similar message [10451.451107] Lustre: Skipped 1 previous similar message [10456.545498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10462.176419] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10462.511223] Lustre: server umount lustre-MDT0000 complete [10465.346477] LustreError: 321829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325545 with bad export cookie 10918528195565891407 [10465.347823] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10465.352483] LustreError: 321829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10465.641383] Lustre: server umount lustre-MDT0001 complete [10472.275724] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10474.282574] Key type lgssc unregistered [10474.514379] LNet: 328133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10474.527122] LNetError: 328133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10474.541101] LNet: Removed LNI 192.168.201.108@tcp [10475.222556] Key type .llcrypt unregistered [10475.223967] Key type ._llcrypt unregistered [10492.371686] Key type ._llcrypt registered [10492.373443] Key type .llcrypt registered [10492.472604] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10493.586015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10493.684221] alg: No test for adler32 (adler32-zlib) [10494.770416] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [10495.043146] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10496.727156] Key type lgssc registered [10497.800193] Lustre: Echo OBD driver; http://www.lustre.org/ [10503.898922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10503.918800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.332283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10508.746397] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10514.914947] LustreError: 329531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.927325] LustreError: 329531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10515.570377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.955654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10518.998952] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10524.287796] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10528.027830] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.022315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.024789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10530.036734] Lustre: Skipped 1 previous similar message [10535.395542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10535.398794] Lustre: Skipped 1 previous similar message [10536.381310] Lustre: server umount lustre-MDT0000 complete [10539.141195] LustreError: 329511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325618 with bad export cookie 11060061190646596988 [10539.152191] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10539.153052] LustreError: 329511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10539.495794] Lustre: server umount lustre-MDT0001 complete [10551.061300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.326874] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10551.355064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10551.533341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.933638] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10561.021691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.107499] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10561.129084] Lustre: 332116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10561.341443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10564.565452] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10569.743239] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10573.569982] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10581.879902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10582.066292] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10582.078445] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10582.085307] LustreError: 333999:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10582.090649] LustreError: 333999:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10582.096970] LustreError: 333999:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10582.159753] Lustre: server umount lustre-OST0000 complete [10582.163432] LustreError: 333999:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10590.503056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10590.640085] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10590.794531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10595.161953] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10600.577335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10600.688754] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10606.677788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10606.753208] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10606.835921] Lustre: lustre-OST0001: new disk, initializing [10606.840758] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10606.851763] Lustre: 335538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10606.923338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10608.004622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10608.010654] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10608.130764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10610.989963] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10616.238689] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10618.336261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10618.342047] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.346265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10618.359283] Lustre: Skipped 1 previous similar message [10623.466051] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10623.470054] Lustre: Skipped 2 previous similar messages [10624.425204] Lustre: server umount lustre-OST0001 complete [10628.577072] LustreError: 335634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.579647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.580376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10628.595927] LustreError: 335634:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10628.618979] Lustre: Skipped 1 previous similar message [10633.703274] LustreError: 334466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.709981] LustreError: 334466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10638.821112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10638.821768] LustreError: 334466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.826023] Lustre: Skipped 4 previous similar messages [10638.847250] LustreError: 334466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10641.375189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10641.491518] Lustre: server umount lustre-OST0000 complete [10643.947042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10648.551130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10648.559178] LustreError: Skipped 1 previous similar message [10648.563805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.271162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10658.410122] Lustre: server umount lustre-MDT0000 complete [10659.297590] LustreError: 333376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10659.309748] LustreError: 333376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10661.323023] LustreError: 332107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325741 with bad export cookie 11060061190646597541 [10661.329067] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10661.337824] LustreError: 332107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10661.989733] Lustre: server umount lustre-MDT0001 complete [10670.250795] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [10672.266649] Key type lgssc unregistered [10672.484671] LNet: 337499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10672.489556] LNetError: 337499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10672.501090] LNet: Removed LNI 192.168.201.108@tcp [10673.124131] Key type .llcrypt unregistered [10673.125544] Key type ._llcrypt unregistered [10727.090887] Key type ._llcrypt registered [10727.094587] Key type .llcrypt registered [10727.173391] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [10737.669936] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [10738.539571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10738.571315] alg: No test for adler32 (adler32-zlib) [10739.646572] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [10739.801994] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [10741.455143] Key type lgssc registered [10742.488072] Lustre: Echo OBD driver; http://www.lustre.org/ [10766.796161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10766.814698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.016627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10768.053430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10768.137542] Lustre: lustre-MDT0000: new disk, initializing [10768.206230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10768.219963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10771.250136] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10779.430606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.539489] Lustre: 342412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10779.620622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10779.626313] Lustre: Skipped 1 previous similar message [10779.724283] Lustre: lustre-MDT0001: new disk, initializing [10779.812563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10779.851198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10779.866930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10783.188495] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10790.934422] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10794.855705] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10799.436259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10799.683683] Lustre: lustre-OST0000: new disk, initializing [10799.686845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10799.692881] Lustre: 344378:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10799.769238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10803.585609] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10805.803427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10805.817066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10805.879647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10809.890534] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10812.870777] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10813.015844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10815.958411] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10816.113362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10821.093698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10821.098708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10821.112664] Lustre: Skipped 1 previous similar message [10821.126743] Lustre: Skipped 1 previous similar message [10823.735978] Lustre: server umount lustre-OST0000 complete [10831.332241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10831.343723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10831.344682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10831.354910] Lustre: Skipped 1 previous similar message [10832.686567] Lustre: server umount lustre-MDT0000 complete [10835.253250] LustreError: 342403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785325914 with bad export cookie 1450127422707773809 [10835.260629] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10835.264836] LustreError: 342403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10835.835140] Lustre: server umount lustre-MDT0001 complete [10841.435972] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:51:59 (1785325919) [10846.368060] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:52:04 (1785325924) [10851.617813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.136961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10855.319553] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10857.446421] LustreError: 347130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.468178] LustreError: 347130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10860.997817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.360518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10864.069759] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10868.851489] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10873.234471] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10877.972604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10878.271463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10882.039683] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10887.111748] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10903.135872] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10904.339818] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10905.411458] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10906.378594] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10907.616407] Lustre: Failing over lustre-MDT0000 [10907.623544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [10907.629434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10907.636344] Lustre: Skipped 1 previous similar message [10907.893185] Lustre: server umount lustre-MDT0000 complete [10908.131342] LustreError: 347130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10908.141507] LustreError: 347130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10913.248653] LustreError: 349619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.262654] LustreError: 349619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10918.368076] LustreError: 347129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.374316] LustreError: 347129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10923.489316] Lustre: 340637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785325987/real 1785325987] req@00000000396844f3 x1872049806047872/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785326003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10923.495426] LustreError: 347126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.502897] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.520467] LustreError: 347126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10923.714645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.727876] LustreError: 340634:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ca4b5f32 x1872049806052608/t0(0) o250->MGC192.168.201.108@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10933.755169] LustreError: 347126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.769342] LustreError: 347126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10933.989928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10934.038910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10934.439655] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10936.715630] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10939.368974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10939.388388] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10939.442593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10942.860231] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10943.795966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10946.251335] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10947.313303] Lustre: Failing over lustre-MDT0000 [10947.470818] Lustre: server umount lustre-MDT0000 complete [10949.603833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10949.606840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10949.610081] Lustre: Skipped 2 previous similar messages [10949.810071] LustreError: 350734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10949.818280] LustreError: 350734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10962.562707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10962.626494] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10962.801950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10965.011148] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [10965.150164] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10968.039902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10968.047753] Lustre: Skipped 2 previous similar messages [10968.068889] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10968.109681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10971.688972] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10972.523188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10978.272061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10978.274795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.284099] Lustre: Skipped 2 previous similar messages [10978.303320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.309397] Lustre: Skipped 1 previous similar message [10983.399771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10983.403798] Lustre: Skipped 1 previous similar message [10983.618831] Lustre: server umount lustre-MDT0000 complete [10985.755764] LustreError: 347111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326065 with bad export cookie 1450127422707776749 [10985.758534] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10985.761161] LustreError: 347111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10985.954495] Lustre: server umount lustre-MDT0001 complete [10998.343137] Lustre: server umount lustre-OST0000 complete [11008.151870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11008.606248] LustreError: 353490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.618934] LustreError: 353490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11008.673948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11008.680253] Lustre: Skipped 1 previous similar message [11011.377243] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11016.669295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11019.774266] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11024.328384] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11027.878373] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11032.024349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11034.292437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11035.660203] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11040.107445] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11049.951914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11049.960558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.976044] Lustre: Skipped 2 previous similar messages [11049.979408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11049.985570] Lustre: Skipped 1 previous similar message [11060.198900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11060.204362] Lustre: Skipped 5 previous similar messages [11062.239142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11062.349033] Lustre: server umount lustre-OST0000 complete [11079.135071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11079.347747] Lustre: server umount lustre-MDT0000 complete [11080.678019] LustreError: 353491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.687777] LustreError: 353491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11081.875368] LustreError: 353472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326161 with bad export cookie 1450127422707777491 [11081.879238] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11081.883233] LustreError: 353472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11082.149684] Lustre: server umount lustre-MDT0001 complete [11089.872743] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11091.657578] Key type lgssc unregistered [11091.860568] LNet: 357021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11091.864864] LNetError: 357021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11091.877601] LNet: Removed LNI 192.168.201.108@tcp [11092.507579] Key type .llcrypt unregistered [11092.509030] Key type ._llcrypt unregistered [11108.181137] Key type ._llcrypt registered [11108.182448] Key type .llcrypt registered [11108.239842] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11109.001574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11109.016849] alg: No test for adler32 (adler32-zlib) [11109.987839] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [11110.219288] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11111.890052] Key type lgssc registered [11112.767468] Lustre: Echo OBD driver; http://www.lustre.org/ [11117.631839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11117.653665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11119.047917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11121.729245] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11126.946832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11127.307986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11129.895953] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11134.953606] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11138.440296] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11143.038373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11143.336875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11144.370436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [11147.016875] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11151.583623] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11162.085256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.089222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.092196] Lustre: Skipped 2 previous similar messages [11162.096067] Lustre: Skipped 1 previous similar message [11165.905312] Lustre: server umount lustre-MDT0000 complete [11167.204139] LustreError: 358511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11167.215442] LustreError: 358511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11168.110692] LustreError: 358495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326247 with bad export cookie 1099143501587243396 [11168.124903] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.126787] LustreError: 358495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11168.363684] Lustre: server umount lustre-MDT0001 complete [11181.045755] Lustre: server umount lustre-OST0000 complete [11189.998703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.495311] LustreError: 362221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11190.567642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.350672] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11195.873923] LustreError: 362222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11199.176184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.490280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11202.455691] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11207.044966] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11210.945962] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11215.069209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11215.362685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11216.437070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [11219.765662] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11224.691599] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11234.786654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.789038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11234.798360] Lustre: Skipped 2 previous similar messages [11234.801986] Lustre: Skipped 2 previous similar messages [11239.844695] Lustre: server umount lustre-MDT0000 complete [11239.906212] LustreError: 362222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.911623] LustreError: 362222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11242.120900] LustreError: 362202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326321 with bad export cookie 1099143501587244509 [11242.131753] LustreError: 362202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11242.143296] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11242.682215] Lustre: server umount lustre-MDT0001 complete [11254.611815] Lustre: server umount lustre-OST0000 complete [11263.894936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11264.329746] LustreError: 366130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.387803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11267.077888] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11272.568506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.243386] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11279.733921] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11283.251309] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11287.822892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11290.157672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [11292.222688] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11296.900806] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11303.393871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11303.395852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11303.407097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11303.407101] Lustre: Skipped 1 previous similar message [11303.412232] Lustre: Skipped 1 previous similar message [11307.013202] Lustre: server umount lustre-OST0000 complete [11313.633043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11313.637218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11313.644105] Lustre: Skipped 1 previous similar message [11313.651014] Lustre: Skipped 2 previous similar messages [11317.757840] Lustre: server umount lustre-MDT0000 complete [11318.758461] LustreError: 366125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11318.774391] LustreError: 366125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11320.261624] LustreError: 366112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326399 with bad export cookie 1099143501587245531 [11320.265558] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11320.274431] LustreError: 366112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11320.701520] Lustre: server umount lustre-MDT0001 complete [11326.942230] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11328.710761] Key type lgssc unregistered [11328.902616] LNet: 369660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11328.905221] LNetError: 369660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11328.915715] LNet: Removed LNI 192.168.201.108@tcp [11329.490158] Key type .llcrypt unregistered [11329.491613] Key type ._llcrypt unregistered [11344.304693] Key type ._llcrypt registered [11344.306282] Key type .llcrypt registered [11344.356127] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11344.945736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11345.006161] alg: No test for adler32 (adler32-zlib) [11345.906662] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [11346.033315] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11347.631114] Key type lgssc registered [11348.505648] Lustre: Echo OBD driver; http://www.lustre.org/ [11353.980300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11353.995261] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.999667] LustreError: 371079:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11354.014455] LustreError: 371079:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11354.016967] LustreError: 371079:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11354.019796] LustreError: 371079:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11354.023175] LustreError: 371079:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11358.778467] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11358.782659] LustreError: 371476:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11358.795663] LustreError: 371476:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11358.802664] LustreError: 371476:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11358.809220] LustreError: 371476:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11358.814270] LustreError: 371476:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11370.870151] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11372.640030] Key type lgssc unregistered [11372.846339] LNet: 372478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11372.852269] LNetError: 372478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11372.877281] LNet: Removed LNI 192.168.201.108@tcp [11373.465340] Key type .llcrypt unregistered [11373.466660] Key type ._llcrypt unregistered [11385.349636] Key type ._llcrypt registered [11385.353480] Key type .llcrypt registered [11385.446404] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [11394.198605] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11394.957408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11395.002228] alg: No test for adler32 (adler32-zlib) [11395.998802] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [11396.209714] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11397.863397] Key type lgssc registered [11398.855263] Lustre: Echo OBD driver; http://www.lustre.org/ [11419.651963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11419.667937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11420.898554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11420.933716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11421.021343] Lustre: lustre-MDT0000: new disk, initializing [11421.060059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11421.067842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11423.592539] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11430.422184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.518555] Lustre: 375622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11430.570579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11430.579392] Lustre: Skipped 1 previous similar message [11430.654873] Lustre: lustre-MDT0001: new disk, initializing [11430.723718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11430.760502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11430.769489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11433.460593] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11439.662861] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11443.187322] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11447.081662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11447.311619] Lustre: lustre-OST0000: new disk, initializing [11447.316606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11447.323208] Lustre: 377588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11447.400434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11451.011748] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11453.507057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11453.517545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11453.595015] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11456.999692] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11459.283864] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11459.381877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11462.061719] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11462.209537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11464.164631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11464.166328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11464.174575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11464.181200] Lustre: Skipped 1 previous similar message [11464.194615] Lustre: Skipped 1 previous similar message [11469.290537] Lustre: server umount lustre-OST0000 complete [11471.844568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11471.849524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.857229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11474.401106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11474.404056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.942937] Lustre: server umount lustre-MDT0000 complete [11480.101464] LustreError: 377587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326559 with bad export cookie 7526869809851957276 [11480.109892] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11480.478890] Lustre: server umount lustre-MDT0001 complete [11485.496280] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11486.470521] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:02:44 (1785326564) [11490.234636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11490.485536] LustreError: 380043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11490.496493] LustreError: 380043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11490.536232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11492.989282] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11495.905584] LustreError: 380044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11498.357765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.638900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11501.387959] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11505.787553] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11508.745990] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11512.615566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11512.866246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11516.556312] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11520.658708] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11524.414405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11524.519507] Lustre: lustre-OST0001: new disk, initializing [11524.522401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11524.526460] Lustre: 382877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11527.921303] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11530.773028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11530.779061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11530.807240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11533.774836] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11540.138686] Lustre: lustre-OST0001: Not available for connect from 192.168.201.8@tcp (stopping) [11540.142922] Lustre: Skipped 1 previous similar message [11540.966393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11540.967876] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11540.977444] Lustre: Skipped 1 previous similar message [11542.231905] Lustre: server umount lustre-OST0001 complete [11546.092698] LustreError: 382537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11546.099668] LustreError: 382537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11551.211360] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.222950] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11556.328228] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.334330] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11556.832486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11556.838185] LustreError: Skipped 1 previous similar message [11556.840532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11556.849261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11556.852546] Lustre: Skipped 2 previous similar messages [11566.566981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11566.569534] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.571941] Lustre: Skipped 4 previous similar messages [11566.589902] LustreError: 381862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11571.167490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11571.300172] Lustre: server umount lustre-OST0000 complete [11575.274378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11575.281402] LustreError: Skipped 1 previous similar message [11575.284705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.294335] Lustre: Skipped 1 previous similar message [11579.763648] Lustre: server umount lustre-MDT0000 complete [11581.729977] LustreError: 380025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326661 with bad export cookie 7526869809851958158 [11581.737757] LustreError: 380025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11581.741893] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11596.255166] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11596.485079] Lustre: server umount lustre-MDT0001 complete [11602.271099] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11603.963922] Key type lgssc unregistered [11604.125577] LNet: 384864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11604.129720] LNetError: 384864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11604.138623] LNet: Removed LNI 192.168.201.108@tcp [11604.684554] Key type .llcrypt unregistered [11604.686482] Key type ._llcrypt unregistered [11632.661064] Key type ._llcrypt registered [11632.664286] Key type .llcrypt registered [11632.736152] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11633.486522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11633.511996] alg: No test for adler32 (adler32-zlib) [11634.511718] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [11634.676252] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11636.295152] Key type lgssc registered [11637.022946] Lustre: Echo OBD driver; http://www.lustre.org/ [11641.941179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11641.955616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.179475] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11643.193493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11643.312886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11646.190250] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11651.514520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.600595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11651.624120] Lustre: 387640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11651.839184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11654.755541] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11659.479823] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11662.594182] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11667.429235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11667.435087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.450717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.937153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11673.059965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11673.066839] Lustre: Skipped 2 previous similar messages [11678.175133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11678.176203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.194502] Lustre: Skipped 1 previous similar message [11678.403454] Lustre: server umount lustre-MDT0000 complete [11680.850314] LustreError: 387630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326760 with bad export cookie 13867736810806114141 [11680.857322] LustreError: 387630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11680.860084] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11681.177531] Lustre: server umount lustre-MDT0001 complete [11686.771263] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:06:04 (1785326764) [11690.164565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.460575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11693.119565] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11695.584173] LustreError: 390149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.596332] LustreError: 390149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11698.273815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.550759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11701.095509] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11705.328334] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11708.069240] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11711.730121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11711.893041] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11712.068377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11715.771132] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11719.912712] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11729.919413] Lustre: Permanently deactivating lustre-OST0000 [11729.931879] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11736.082538] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11737.105552] LustreError: 393134:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11737.114529] LustreError: 393134:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11737.120563] LustreError: 393134:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11738.520400] LustreError: 393182:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11738.526482] LustreError: 393182:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11746.058735] Lustre: server umount lustre-OST0000 complete [11747.937806] LustreError: 393431:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11747.942641] LustreError: 393431:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11749.856264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11749.859796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.882618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11752.417787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11757.540932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11757.548806] Lustre: Skipped 3 previous similar messages [11758.622343] Lustre: server umount lustre-MDT0000 complete [11760.847028] LustreError: 390131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326840 with bad export cookie 13867736810806114708 [11760.847641] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11760.852391] LustreError: 390131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11770.446781] Lustre: server umount lustre-MDT0001 complete [11775.237720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.897921] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11779.695117] LustreError: 394590:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11783.559603] Lustre: 394929:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.108@tcp [11784.908733] Lustre: server umount MGS complete [11791.609895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.958658] LustreError: 395651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11791.980618] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11791.982969] Lustre: Skipped 1 previous similar message [11792.035063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11794.356843] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11797.481712] LustreError: 395652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.494925] LustreError: 395652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11799.280864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.437220] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11801.816989] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11806.191232] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11809.526780] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11810.644923] Lustre: Permanently reactivating lustre-OST0000 [11810.660335] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11810.664932] Lustre: Skipped 2 previous similar messages [11831.775168] LustreError: 397285:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11835.934163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11836.154434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11836.156792] Lustre: Skipped 1 previous similar message [11839.099231] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11843.059690] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11851.746721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11851.749180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.750733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11851.757506] LustreError: Skipped 1 previous similar message [11851.775791] Lustre: Skipped 1 previous similar message [11852.394404] Lustre: server umount lustre-OST0000 complete [11854.823823] Lustre: server umount lustre-MDT0000 complete [11855.842810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11855.848986] LustreError: 399058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.717869] LustreError: 395634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785326936 with bad export cookie 13867736810806115758 [11856.723457] LustreError: 395634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11856.726538] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11856.737141] Lustre: 395645:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.108@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [11856.949108] Lustre: server umount lustre-MDT0001 complete [11862.619228] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [11864.150482] Key type lgssc unregistered [11864.365721] LNet: 400009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11864.371920] LNetError: 400009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11864.387099] LNet: Removed LNI 192.168.201.108@tcp [11865.005566] Key type .llcrypt unregistered [11865.007933] Key type ._llcrypt unregistered [11876.354114] Key type ._llcrypt registered [11876.356381] Key type .llcrypt registered [11876.427936] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [11884.830690] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [11885.485186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11885.495195] alg: No test for adler32 (adler32-zlib) [11886.421831] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [11886.568780] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [11888.183172] Key type lgssc registered [11888.824903] Lustre: Echo OBD driver; http://www.lustre.org/ [11907.210317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11907.242394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.385132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11908.407549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11908.462273] Lustre: lustre-MDT0000: new disk, initializing [11908.518258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11908.530801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11910.617178] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11917.028940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.130965] Lustre: 403143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11917.175620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11917.178353] Lustre: Skipped 1 previous similar message [11917.228827] Lustre: lustre-MDT0001: new disk, initializing [11917.293742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11917.325374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11917.330760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11919.579058] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11925.499072] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11928.740515] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11932.327346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11932.487484] Lustre: lustre-OST0000: new disk, initializing [11932.489173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11932.493293] Lustre: 405110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11932.533996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11936.002320] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11938.825225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11938.829993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11938.896502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11942.049315] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11944.766615] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11944.871483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11947.094392] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11947.223490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11949.026893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.029213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11949.042476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.055311] Lustre: Skipped 1 previous similar message [11954.150905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11954.499376] Lustre: server umount lustre-OST0000 complete [11958.239990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11958.245411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11958.250078] Lustre: Skipped 1 previous similar message [11958.253065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11958.256956] Lustre: Skipped 1 previous similar message [11959.267067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.384526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.387552] Lustre: Skipped 3 previous similar messages [11969.504539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.508285] Lustre: Skipped 1 previous similar message [11971.039116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11971.145198] Lustre: server umount lustre-MDT0000 complete [11973.420482] LustreError: 403136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327053 with bad export cookie 8452929021149293181 [11973.427307] LustreError: 403136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11973.427726] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11973.626786] Lustre: server umount lustre-MDT0001 complete [11978.147368] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11978.961694] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:10:57 (1785327057) [11982.547447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11982.965652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11985.281393] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11988.452909] LustreError: 407570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.465557] LustreError: 407570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11990.133681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.374492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11992.783652] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [11996.704622] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11999.893021] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12003.544339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12003.756785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.192079] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12011.421652] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12012.596545] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12014.487187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12023.264188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12023.271769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12023.284198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12028.542860] Lustre: server umount lustre-OST0000 complete [12031.460583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12031.460990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12031.473645] Lustre: Skipped 1 previous similar message [12041.697034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.701038] Lustre: Skipped 7 previous similar messages [12045.279771] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12045.437072] Lustre: server umount lustre-MDT0000 complete [12046.816155] LustreError: 407570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12046.825494] LustreError: 407570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12047.686437] LustreError: 407551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327127 with bad export cookie 8452929021149294063 [12047.689350] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12047.695234] LustreError: 407551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12048.257980] Lustre: server umount lustre-MDT0001 complete [12054.091761] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12055.837290] Key type lgssc unregistered [12056.053086] LNet: 411290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12056.061515] LNetError: 411290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12056.078148] LNet: Removed LNI 192.168.201.108@tcp [12056.634064] Key type .llcrypt unregistered [12056.635488] Key type ._llcrypt unregistered [12069.924393] Key type ._llcrypt registered [12069.926448] Key type .llcrypt registered [12069.985363] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12070.107180] Key type .llcrypt unregistered [12070.108422] Key type ._llcrypt unregistered [12080.211642] Key type ._llcrypt registered [12080.214696] Key type .llcrypt registered [12080.272242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12080.968759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12081.022740] alg: No test for adler32 (adler32-zlib) [12081.974154] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12082.172628] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12083.831145] Key type lgssc registered [12084.753550] Lustre: Echo OBD driver; http://www.lustre.org/ [12089.697359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12089.720854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.186167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.555103] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12098.781520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12099.082598] LustreError: 413479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.095731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.098986] LustreError: 413479:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12101.159261] LustreError: 413480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12101.173435] LustreError: 413480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12101.186517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12102.924848] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12106.211535] LustreError: 413475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.224592] LustreError: 413475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12106.883590] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12110.916182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12111.114402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12113.815655] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12126.692209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12126.696397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12126.704097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12126.708948] Lustre: Skipped 1 previous similar message [12128.731535] Lustre: server umount lustre-OST0000 complete [12131.809906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.810078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12131.825205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12131.825991] Lustre: Skipped 2 previous similar messages [12131.837102] LustreError: Skipped 2 previous similar messages [12133.048113] Lustre: server umount lustre-MDT0000 complete [12135.144927] LustreError: 413460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327214 with bad export cookie 15912678149081435432 [12135.154969] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.345573] Lustre: server umount lustre-MDT0001 complete [12141.467582] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12143.005393] Key type lgssc unregistered [12143.179624] LNet: 416621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12143.184415] LNetError: 416621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12143.194759] LNet: Removed LNI 192.168.201.108@tcp [12143.674840] Key type .llcrypt unregistered [12143.676080] Key type ._llcrypt unregistered [12157.188823] Key type ._llcrypt registered [12157.192473] Key type .llcrypt registered [12157.303932] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12157.942799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12158.011423] alg: No test for adler32 (adler32-zlib) [12158.979935] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12159.144592] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12160.767199] Key type lgssc registered [12161.421071] Lustre: Echo OBD driver; http://www.lustre.org/ [12166.312331] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12166.331731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12181.855487] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12186.975315] LustreError: 417990:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [12213.663213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.020942] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12220.985329] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12224.675487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.058501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12225.100713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12227.515142] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12232.241385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12232.422484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12234.637777] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12238.800514] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12241.754948] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12248.032486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12248.036931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.040138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.046432] Lustre: Skipped 1 previous similar message [12252.300164] Lustre: server umount lustre-OST0000 complete [12255.716463] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12255.720414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.735386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.740310] Lustre: Skipped 1 previous similar message [12258.275290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12258.279705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.630976] Lustre: server umount lustre-MDT0000 complete [12262.692802] LustreError: 417998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327342 with bad export cookie 4009800760508204856 [12262.700141] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12262.701897] LustreError: 417998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12262.876145] Lustre: server umount lustre-MDT0001 complete [12268.847884] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12270.412906] Key type lgssc unregistered [12270.584673] LNet: 421558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12270.587917] LNetError: 421558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12270.595865] LNet: Removed LNI 192.168.201.108@tcp [12271.095235] Key type .llcrypt unregistered [12271.096901] Key type ._llcrypt unregistered [12284.701583] Key type ._llcrypt registered [12284.703502] Key type .llcrypt registered [12284.748861] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12285.498604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12285.525111] alg: No test for adler32 (adler32-zlib) [12286.467857] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12286.645158] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12288.255251] Key type lgssc registered [12288.891617] Lustre: Echo OBD driver; http://www.lustre.org/ [12293.239653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12293.250504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12294.601972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12296.854093] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12301.271715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.500494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12303.847559] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12307.653100] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12310.461452] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12314.262385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12314.429676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12317.755930] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12319.537824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12322.295683] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12324.833488] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12324.841120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12326.880730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12326.896537] Lustre: Skipped 3 previous similar messages [12329.138478] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [12331.020657] Lustre: server umount lustre-MDT0000 complete [12332.001689] LustreError: 422947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.007563] LustreError: 422947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12332.402430] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.8@tcp arrived at 1785327412 with bad export cookie 17288283615674917774 [12332.414735] LustreError: 422934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12333.481375] LustreError: 422933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327413 with bad export cookie 17288283615674916927 [12333.489316] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12333.490756] LustreError: 422933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12337.120022] LustreError: 422952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.120763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.129739] LustreError: 422952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12337.144415] Lustre: Skipped 2 previous similar messages [12337.148983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12338.147609] LustreError: 422953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.160970] LustreError: 422953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12339.615326] Lustre: server umount lustre-OST0000 complete [12343.287210] LustreError: 423681:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c5af4488 x1872051428156672/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12343.386929] Lustre: server umount lustre-MDT0001 complete [12350.271703] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12352.092660] Key type lgssc unregistered [12352.295279] LNet: 426630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12352.308208] LNetError: 426630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12353.381982] LNet: Removed LNI 192.168.201.108@tcp [12353.957520] Key type .llcrypt unregistered [12353.959341] Key type ._llcrypt unregistered [12367.621765] Key type ._llcrypt registered [12367.623105] Key type .llcrypt registered [12367.687173] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12368.256963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12368.350116] alg: No test for adler32 (adler32-zlib) [12369.280349] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12369.416936] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12371.031150] Key type lgssc registered [12371.622683] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.781713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.795977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12377.039563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12379.326743] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12383.873778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.064850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12386.256470] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12390.507550] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12393.603618] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12397.467663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12397.774115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12401.083627] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12401.908976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12404.927520] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12412.388523] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.396037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12414.649685] Lustre: server umount lustre-MDT0001 complete [12417.506863] LustreError: 430511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.518368] LustreError: 430511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12422.625336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12422.627443] LustreError: 428024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.634385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12422.634394] Lustre: Skipped 1 previous similar message [12422.635835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12422.658392] Lustre: Skipped 2 previous similar messages [12423.783442] Lustre: server umount lustre-OST0000 complete [12426.159215] Lustre: server umount lustre-MDT0000 complete [12433.237238] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12434.627455] Key type lgssc unregistered [12434.770864] LNet: 431702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12434.774217] LNetError: 431702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12434.782832] LNet: Removed LNI 192.168.201.108@tcp [12435.270695] Key type .llcrypt unregistered [12435.272182] Key type ._llcrypt unregistered [12448.127902] Key type ._llcrypt registered [12448.129303] Key type .llcrypt registered [12448.183319] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [12456.145104] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12456.663345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12456.691104] alg: No test for adler32 (adler32-zlib) [12457.595474] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12457.739797] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12459.359456] Key type lgssc registered [12459.865696] Lustre: Echo OBD driver; http://www.lustre.org/ [12478.363182] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12486.030806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12486.078597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12487.200221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12487.213655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12487.260875] Lustre: lustre-MDT0000: new disk, initializing [12487.302144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12487.316741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12489.629744] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12498.121597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12498.290673] Lustre: lustre-OST0000: new disk, initializing [12498.293417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12498.295464] Lustre: Skipped 1 previous similar message [12498.298457] Lustre: 436711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12498.337730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12499.307115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12499.314207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12499.330721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12501.285984] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12506.368871] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12515.766357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.843292] Lustre: 435691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12515.881664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12515.927074] Lustre: lustre-MDT0001: new disk, initializing [12516.003626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12516.028758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12516.035090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12518.402494] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12523.835711] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12523.971520] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12526.469443] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12526.597449] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12530.362932] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [12533.222388] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12533.222673] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12533.223407] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12533.240536] Lustre: Skipped 1 previous similar message [12535.259773] Lustre: server umount lustre-MDT0001 complete [12543.464717] LustreError: 436718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.474153] LustreError: 436718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12548.584058] LustreError: 435698:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.602261] LustreError: 435698:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12550.625499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12550.627940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12550.642716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12550.651669] Lustre: Skipped 1 previous similar message [12553.696973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12553.701833] LustreError: 435702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12555.523607] Lustre: server umount lustre-OST0000 complete [12563.978933] Lustre: server umount lustre-MDT0000 complete [12571.898336] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12573.503555] Key type lgssc unregistered [12573.693474] LNet: 440453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12573.698641] LNetError: 440453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12573.711366] LNet: Removed LNI 192.168.201.108@tcp [12574.170513] Key type .llcrypt unregistered [12574.172159] Key type ._llcrypt unregistered [12594.341601] Key type ._llcrypt registered [12594.342610] Key type .llcrypt registered [12594.399474] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [12594.546300] Key type .llcrypt unregistered [12594.550385] Key type ._llcrypt unregistered [12603.546408] Key type ._llcrypt registered [12603.547631] Key type .llcrypt registered [12603.603090] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [12604.054618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12604.087890] alg: No test for adler32 (adler32-zlib) [12605.016292] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [12605.158912] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [12606.751148] Key type lgssc registered [12607.527571] Lustre: Echo OBD driver; http://www.lustre.org/ [12621.169287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12621.192723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12622.315218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12622.332515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12622.388542] Lustre: lustre-MDT0000: new disk, initializing [12622.428630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12622.438569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12624.752574] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12630.465609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12630.528578] Lustre: 443806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12630.558854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12630.560490] Lustre: Skipped 1 previous similar message [12630.599752] Lustre: lustre-MDT0001: new disk, initializing [12630.647608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12630.671013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12630.678887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12633.173529] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12638.566902] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12641.340392] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12644.797496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12644.971567] Lustre: lustre-OST0000: new disk, initializing [12644.976894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12644.979915] Lustre: 445771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12645.054027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12647.976383] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12650.513138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12650.530709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12650.590156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12653.702433] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12660.707511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12660.714320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12660.715583] Lustre: Skipped 1 previous similar message [12660.724408] Lustre: Skipped 2 previous similar messages [12663.675197] Lustre: server umount lustre-MDT0000 complete [12665.735417] LustreError: 443799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327745 with bad export cookie 4010902959138496428 [12665.736727] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12665.748176] LustreError: 443799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12665.829268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12665.837765] Lustre: Skipped 1 previous similar message [12665.910470] Lustre: server umount lustre-MDT0001 complete [12673.990769] Lustre: server umount lustre-OST0000 complete [12679.379418] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:22:37 (1785327757) [12684.154731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.438361] LustreError: 447795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12684.453179] LustreError: 447795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12684.483631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12686.855203] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12689.888322] LustreError: 447796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12691.716619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12691.904051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12694.030308] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12697.913420] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12700.858476] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12717.414263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12717.590699] Lustre: Found index 0 for lustre-OST0000, updating log [12717.745023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12720.895436] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [12724.611914] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12731.056598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12738.593735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12738.598588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12739.617871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12739.622629] Lustre: Skipped 1 previous similar message [12741.667563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12741.673573] Lustre: Skipped 1 previous similar message [12743.750286] LustreError: 450406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.787907] Lustre: server umount lustre-OST0000 complete [12758.496544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12758.502980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12758.512980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12758.515299] Lustre: Skipped 1 previous similar message [12765.496585] Lustre: server umount lustre-MDT0000 complete [12766.179683] LustreError: 448862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.739658] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785327847 with bad export cookie 4010902959138497807 [12767.741553] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12767.749615] LustreError: 447776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12767.919171] Lustre: server umount lustre-MDT0001 complete [12772.068641] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:24:10 (1785327850) [12779.231457] LNet: 1 local NIs in recovery (showing 1): 192.168.201.108@tcp [13364.004219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13364.249762] LustreError: 454218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13364.267342] LustreError: 454218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13364.309275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13366.583553] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13370.768536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13370.944225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13373.204171] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13377.316634] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13380.299804] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13397.094939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13397.217516] Lustre: Found index 0 for lustre-OST0000, updating log [13397.377131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13400.716885] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13405.136936] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13408.746192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13408.751088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13408.756331] Lustre: Skipped 1 previous similar message [13408.759061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13408.762235] Lustre: Skipped 2 previous similar messages [13412.353348] Lustre: server umount lustre-OST0000 complete [13416.417557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.430512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13416.466588] Lustre: Skipped 1 previous similar message [13418.207411] LNet: 1 local NIs in recovery (showing 1): 192.168.201.108@tcp [13418.529347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13418.533695] Lustre: Skipped 4 previous similar messages [13421.715543] Lustre: server umount lustre-MDT0000 complete [13422.627595] LustreError: 454215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.657462] LustreError: 454215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13425.950817] LustreError: 454198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785328505 with bad export cookie 4010902959138498878 [13425.951937] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13426.000590] LustreError: 454198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13426.213385] Lustre: server umount lustre-MDT0001 complete [13458.915707] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [13481.008713] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [13564.103345] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:37:20 (1785328640) [13593.964452] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [13597.971056] Key type lgssc unregistered [13598.513650] LNet: 460862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13598.549384] LNetError: 460862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13657.965334] LNet: Removed LNI 192.168.201.108@tcp [13658.852103] Key type .llcrypt unregistered [13658.854572] Key type ._llcrypt unregistered [13677.496626] Key type ._llcrypt registered [13677.498217] Key type .llcrypt registered [13677.712062] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [13679.283455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13679.336826] alg: No test for adler32 (adler32-zlib) [13680.506183] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [13680.766406] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [13682.447194] Key type lgssc registered [13683.817648] Lustre: Echo OBD driver; http://www.lustre.org/ [13688.451598] LNet: Added LNI 192.168.201.108@tcp100 [8/256/0/180] [13689.474929] LNet: Added LNI 192.168.201.108@tcp101 [8/256/0/180] [13690.266994] LNet: Added LNI 192.168.201.108@tcp102 [8/256/0/180] [13691.461441] LNet: Added LNI 192.168.201.108@tcp103 [8/256/0/180] [13692.543480] LNet: Added LNI 192.168.201.108@tcp104 [8/256/0/180] [13693.727847] LNet: Added LNI 192.168.201.108@tcp105 [8/256/0/180] [13694.646711] LNet: Added LNI 192.168.201.108@tcp106 [8/256/0/180] [13695.707216] LNet: Added LNI 192.168.201.108@tcp107 [8/256/0/180] [13696.690173] LNet: Added LNI 192.168.201.108@tcp108 [8/256/0/180] [13697.926562] LNet: Added LNI 192.168.201.108@tcp109 [8/256/0/180] [13698.831922] LNet: Added LNI 192.168.201.108@tcp110 [8/256/0/180] [13699.929874] LNet: Added LNI 192.168.201.108@tcp111 [8/256/0/180] [13701.406390] LNet: Added LNI 192.168.201.108@tcp112 [8/256/0/180] [13702.890808] LNet: Added LNI 192.168.201.108@tcp113 [8/256/0/180] [13704.142295] LNet: Added LNI 192.168.201.108@tcp114 [8/256/0/180] [13705.305898] LNet: Added LNI 192.168.201.108@tcp115 [8/256/0/180] [13706.607817] LNet: Added LNI 192.168.201.108@tcp116 [8/256/0/180] [13707.806108] LNet: Added LNI 192.168.201.108@tcp117 [8/256/0/180] [13708.999981] LNet: Added LNI 192.168.201.108@tcp118 [8/256/0/180] [13709.917752] LNet: Added LNI 192.168.201.108@tcp119 [8/256/0/180] [13710.955757] LNet: Added LNI 192.168.201.108@tcp120 [8/256/0/180] [13712.083542] LNet: Added LNI 192.168.201.108@tcp121 [8/256/0/180] [13712.965671] LNet: Added LNI 192.168.201.108@tcp122 [8/256/0/180] [13713.835147] LNet: Added LNI 192.168.201.108@tcp123 [8/256/0/180] [13714.803199] LNet: Added LNI 192.168.201.108@tcp124 [8/256/0/180] [13715.829151] LNet: Added LNI 192.168.201.108@tcp125 [8/256/0/180] [13716.925627] LNet: Added LNI 192.168.201.108@tcp126 [8/256/0/180] [13718.213355] LNet: Added LNI 192.168.201.108@tcp127 [8/256/0/180] [13719.240948] LNet: Added LNI 192.168.201.108@tcp128 [8/256/0/180] [13720.553763] LNet: Added LNI 192.168.201.108@tcp129 [8/256/0/180] [13721.540654] LNet: Added LNI 192.168.201.108@tcp130 [8/256/0/180] [13722.739429] LNet: Added LNI 192.168.201.108@tcp131 [8/256/0/180] [13723.906386] LNet: Added LNI 192.168.201.108@tcp132 [8/256/0/180] [13725.235224] LNet: Added LNI 192.168.201.108@tcp133 [8/256/0/180] [13726.363850] LNet: Added LNI 192.168.201.108@tcp134 [8/256/0/180] [13727.816136] LNet: Added LNI 192.168.201.108@tcp135 [8/256/0/180] [13736.661890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13736.729388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.889502] Lustre: 463888:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13738.016095] LNetError: 461345:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13738.038724] LNetError: Protocol error connecting to 192.168.201.108@tcp117 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13738.148936] LNet: 6 local NIs in recovery (showing 6): 192.168.201.108@tcp124, 192.168.201.108@tcp122, 192.168.201.108@tcp125, 192.168.201.108@tcp126, 192.168.201.108@tcp128, 192.168.201.108@tcp129 [13738.284926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13738.347081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13738.462571] Lustre: lustre-MDT0000: new disk, initializing [13738.628932] Lustre: 463942:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13738.655879] Lustre: 463942:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13738.703544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13738.726990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13739.170662] LNetError: 461346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13739.194799] LNetError: 461346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [13739.227642] LNetError: Protocol error connecting to 192.168.201.108@tcp116 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13739.246541] LNetError: Skipped 15 previous similar messages [13741.282429] LNetError: 461347:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13741.303514] LNetError: 461347:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [13741.315577] LNetError: Protocol error connecting to 192.168.201.108@tcp124 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13741.323738] LNetError: Skipped 4 previous similar messages [13745.377079] LNetError: 461346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13745.387517] LNetError: 461346:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13745.403384] LNetError: Protocol error connecting to 192.168.201.108@tcp124 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13745.413726] LNetError: Skipped 1 previous similar message [13745.842873] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13753.572964] LNetError: 463907:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13753.591913] LNetError: Protocol error connecting to 192.168.201.108@tcp124 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13757.900326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13758.119502] Lustre: 463904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13758.230556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13758.237948] Lustre: Skipped 1 previous similar message [13758.349112] Lustre: lustre-MDT0001: new disk, initializing [13758.440322] Lustre: 464706:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13758.463955] Lustre: 464706:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13758.652502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13758.725810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13758.750874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13764.523319] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13767.231695] Lustre: 465069:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13767.240858] Lustre: 465069:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13768.929630] LNetError: 461345:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13768.953336] LNetError: Protocol error connecting to 192.168.201.108@tcp117 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13775.348579] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13782.096504] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13785.318894] LNetError: 463919:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13785.333591] LNetError: 463919:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [13785.344770] LNetError: Protocol error connecting to 192.168.201.108@tcp123 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13785.364314] LNetError: Skipped 16 previous similar messages [13790.027118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13790.513085] Lustre: lustre-OST0000: new disk, initializing [13790.521074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13790.537908] Lustre: 465877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13790.744386] Lustre: 465882:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13790.755227] Lustre: 465882:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13790.855303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13797.599306] LNet: 1 local NIs in recovery (showing 1): 192.168.201.108@tcp124 [13799.164212] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [13800.534119] Lustre: 466294:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.108@tcp: can't add NID 192.168.201.108@tcp130: rc = -75 [13800.551653] Lustre: 466294:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13800.626220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13800.637039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13800.840240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13810.241251] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13818.091597] LNetError: 463913:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.108 [13818.109767] LNetError: 463913:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 31 previous similar messages [13818.125446] LNetError: Protocol error connecting to 192.168.201.108@tcp114 at host 192.168.201.108:988: is it running a compatible version of Lustre? [13818.143027] LNetError: Skipped 31 previous similar messages [13821.414236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13821.419619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13821.427920] LustreError: Skipped 1 previous similar message [13821.433703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13821.451029] Lustre: Skipped 1 previous similar message [13821.485556] Lustre: Skipped 1 previous similar message [13825.208086] Lustre: server umount lustre-OST0000 complete [13831.657610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13831.665132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13831.673698] Lustre: Skipped 1 previous similar message [13831.693293] Lustre: Skipped 1 previous similar message [13836.781091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13836.793128] Lustre: Skipped 1 previous similar message [13837.319071] Lustre: server umount lustre-MDT0000 complete [13841.924325] LustreError: 463897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785328921 with bad export cookie 16796553966744213255 [13841.929564] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13841.936610] LustreError: 463897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13842.343984] Lustre: server umount lustre-MDT0001 complete [13855.813651] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [13858.659431] Key type lgssc unregistered [13859.070854] LNet: 467793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13859.080193] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13859.086401] LNet: Removed LNI 192.168.201.108@tcp [13860.131869] LNet: Removed LNI 192.168.201.108@tcp100 [13860.133535] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13860.147756] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13860.171851] LNet: Removed LNI 192.168.201.108@tcp101 [13861.215173] LNet: Removed LNI 192.168.201.108@tcp102 [13861.254437] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13861.267034] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13861.269674] LNet: Removed LNI 192.168.201.108@tcp103 [13862.303675] LNet: Removed LNI 192.168.201.108@tcp104 [13863.328417] LNet: Removed LNI 192.168.201.108@tcp105 [13863.333642] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13863.352612] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13863.367295] LNet: Removed LNI 192.168.201.108@tcp106 [13864.415413] LNet: Removed LNI 192.168.201.108@tcp107 [13865.443688] LNet: Removed LNI 192.168.201.108@tcp108 [13866.463628] LNet: Removed LNI 192.168.201.108@tcp109 [13867.487261] LNet: Removed LNI 192.168.201.108@tcp110 [13867.491537] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13867.507759] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13867.521989] LNet: Removed LNI 192.168.201.108@tcp111 [13868.577097] LNet: Removed LNI 192.168.201.108@tcp112 [13869.599157] LNet: Removed LNI 192.168.201.108@tcp113 [13870.623655] LNet: Removed LNI 192.168.201.108@tcp114 [13871.647166] LNet: Removed LNI 192.168.201.108@tcp115 [13872.671174] LNet: Removed LNI 192.168.201.108@tcp116 [13873.696539] LNet: Removed LNI 192.168.201.108@tcp117 [13874.719182] LNet: Removed LNI 192.168.201.108@tcp118 [13875.745264] LNet: Removed LNI 192.168.201.108@tcp119 [13875.756227] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13875.792449] LNetError: 467793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13875.810348] LNet: Removed LNI 192.168.201.108@tcp120 [13876.831374] LNet: Removed LNI 192.168.201.108@tcp121 [13877.855263] LNet: Removed LNI 192.168.201.108@tcp122 [13878.879458] LNet: Removed LNI 192.168.201.108@tcp123 [13878.882353] LNet: Removed LNI 192.168.201.108@tcp124 [13879.905788] LNet: Removed LNI 192.168.201.108@tcp125 [13880.927176] LNet: Removed LNI 192.168.201.108@tcp126 [13882.015206] LNet: Removed LNI 192.168.201.108@tcp127 [13883.042062] LNet: Removed LNI 192.168.201.108@tcp128 [13884.127358] LNet: Removed LNI 192.168.201.108@tcp129 [13884.137299] LNet: Removed LNI 192.168.201.108@tcp130 [13884.152685] LNet: Removed LNI 192.168.201.108@tcp131 [13884.165150] LNet: Removed LNI 192.168.201.108@tcp132 [13884.172708] LNet: Removed LNI 192.168.201.108@tcp133 [13884.191580] LNet: Removed LNI 192.168.201.108@tcp134 [13884.212134] LNet: Removed LNI 192.168.201.108@tcp135 [13885.916203] Key type .llcrypt unregistered [13885.920614] Key type ._llcrypt unregistered [13915.607420] Key type ._llcrypt registered [13915.614322] Key type .llcrypt registered [13915.851421] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [13935.278597] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [13936.637848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13936.715591] alg: No test for adler32 (adler32-zlib) [13937.861494] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [13938.088120] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [13939.719178] Key type lgssc registered [13940.814866] Lustre: Echo OBD driver; http://www.lustre.org/ [14001.866594] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:44:36 (1785329076) [14027.212374] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [14033.204117] Key type lgssc unregistered [14034.254783] LNet: 471892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14034.261534] LNetError: 471892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14034.270962] LNet: Removed LNI 192.168.201.108@tcp [14036.712685] Key type .llcrypt unregistered [14036.823384] Key type ._llcrypt unregistered [14067.647559] Key type ._llcrypt registered [14067.649837] Key type .llcrypt registered [14067.915203] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [14069.900250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14069.919661] alg: No test for adler32 (adler32-zlib) [14071.395384] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [14071.833317] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [14073.528745] Key type lgssc registered [14076.347042] Lustre: Echo OBD driver; http://www.lustre.org/ [14098.279940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14098.307916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14100.889563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14100.997607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14101.242243] Lustre: lustre-MDT0000: new disk, initializing [14101.410348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14101.427752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14111.565930] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14130.078244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14130.267327] Lustre: 473106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14130.360459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14130.362935] Lustre: Skipped 1 previous similar message [14130.451338] Lustre: lustre-MDT0001: new disk, initializing [14130.560980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14130.593618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14130.600802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14136.095812] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14149.880626] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14160.961719] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14171.500331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14172.057505] Lustre: lustre-OST0000: new disk, initializing [14172.064389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14172.074982] Lustre: 475080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14172.198729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14179.488724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14179.499714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14179.719542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14181.877628] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14194.434635] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14199.301742] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing lnet_if_list [14206.159264] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14206.206916] LNet: Added LNI 192.168.201.108@tcp10 [8/256/0/180] [14212.480624] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14212.504781] LNet: Added LNI 192.168.201.108@tcp11 [8/256/0/180] [14217.502613] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14217.528938] LNet: Added LNI 192.168.201.108@tcp12 [8/256/0/180] [14222.568851] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14222.647614] LNet: Added LNI 192.168.201.108@tcp13 [8/256/0/180] [14229.902795] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14229.956865] LNet: Added LNI 192.168.201.108@tcp14 [8/256/0/180] [14238.039547] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14238.096667] LNet: Added LNI 192.168.201.108@tcp15 [8/256/0/180] [14249.909359] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14249.985611] LNet: Added LNI 192.168.201.108@tcp16 [8/256/0/180] [14258.088736] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14258.120896] LNet: Added LNI 192.168.201.108@tcp17 [8/256/0/180] [14266.638669] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14266.708662] LNet: Added LNI 192.168.201.108@tcp18 [8/256/0/180] [14274.293381] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14274.371658] LNet: Added LNI 192.168.201.108@tcp19 [8/256/0/180] [14284.546598] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14284.653369] LNet: Added LNI 192.168.201.108@tcp20 [8/256/0/180] [14297.438086] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14297.483119] LNet: Added LNI 192.168.201.108@tcp21 [8/256/0/180] [14306.754393] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14306.786099] LNet: Added LNI 192.168.201.108@tcp22 [8/256/0/180] [14324.279408] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14324.368449] LNet: Added LNI 192.168.201.108@tcp23 [8/256/0/180] [14337.292970] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14337.394436] LNet: Added LNI 192.168.201.108@tcp24 [8/256/0/180] [14350.423087] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14350.463819] LNet: Added LNI 192.168.201.108@tcp25 [8/256/0/180] [14360.002286] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14360.052268] LNet: Added LNI 192.168.201.108@tcp26 [8/256/0/180] [14373.856661] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14373.930474] LNet: Added LNI 192.168.201.108@tcp27 [8/256/0/180] [14383.269826] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14383.325596] LNet: Added LNI 192.168.201.108@tcp28 [8/256/0/180] [14392.570993] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14392.635197] LNet: Added LNI 192.168.201.108@tcp29 [8/256/0/180] [14403.006813] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14403.078805] LNet: Added LNI 192.168.201.108@tcp30 [8/256/0/180] [14411.805702] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14411.850119] LNet: Added LNI 192.168.201.108@tcp31 [8/256/0/180] [14419.907103] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14419.965961] LNet: Added LNI 192.168.201.108@tcp32 [8/256/0/180] [14427.918612] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14427.963473] LNet: Added LNI 192.168.201.108@tcp33 [8/256/0/180] [14436.887613] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14436.946232] LNet: Added LNI 192.168.201.108@tcp34 [8/256/0/180] [14446.023883] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14446.094336] LNet: Added LNI 192.168.201.108@tcp35 [8/256/0/180] [14455.692601] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14455.708417] LNet: Added LNI 192.168.201.108@tcp36 [8/256/0/180] [14463.701165] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14463.727946] LNet: Added LNI 192.168.201.108@tcp37 [8/256/0/180] [14473.282686] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14473.324495] LNet: Added LNI 192.168.201.108@tcp38 [8/256/0/180] [14483.176578] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14483.223865] LNet: Added LNI 192.168.201.108@tcp39 [8/256/0/180] [14494.883238] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14494.939664] LNet: Added LNI 192.168.201.108@tcp40 [8/256/0/180] [14504.273496] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14504.341623] LNet: Added LNI 192.168.201.108@tcp41 [8/256/0/180] [14515.881480] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14515.941493] LNet: Added LNI 192.168.201.108@tcp42 [8/256/0/180] [14530.345416] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14530.418811] LNet: Added LNI 192.168.201.108@tcp43 [8/256/0/180] [14543.340691] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14543.407514] LNet: Added LNI 192.168.201.108@tcp44 [8/256/0/180] [14556.692909] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14556.727588] LNet: Added LNI 192.168.201.108@tcp45 [8/256/0/180] [14568.894611] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14568.957379] LNet: Added LNI 192.168.201.108@tcp46 [8/256/0/180] [14579.621735] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14579.658864] LNet: Added LNI 192.168.201.108@tcp47 [8/256/0/180] [14592.273995] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14592.369187] LNet: Added LNI 192.168.201.108@tcp48 [8/256/0/180] [14608.189731] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14608.256459] LNet: Added LNI 192.168.201.108@tcp49 [8/256/0/180] [14619.585214] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14619.621639] LNet: Added LNI 192.168.201.108@tcp50 [8/256/0/180] [14653.409517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14653.423971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.454654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14656.486963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14656.493426] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14656.535599] Lustre: Skipped 1 previous similar message [14661.603071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14661.626422] Lustre: Skipped 4 previous similar messages [14666.209283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14666.781435] Lustre: server umount lustre-MDT0000 complete [14671.854056] LustreError: 474273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14671.873872] LustreError: 474273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14672.710801] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785329752 with bad export cookie 14697965335437275458 [14672.728627] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14672.746611] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14673.579957] Lustre: server umount lustre-MDT0001 complete [14693.280095] Lustre: 472378:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785329756/real 1785329756] req@000000004310360e x1872053300072064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785329772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14693.309262] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14696.937319] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [14697.451770] Lustre: 472380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785329761/real 1785329761] req@0000000089932e37 x1872053300072320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785329777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14697.501394] Lustre: 472380:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14698.494052] Lustre: 472378:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785329762/real 1785329762] req@00000000e665c27d x1872053300072576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785329778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14703.584389] Lustre: 472377:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785329767/real 1785329767] req@0000000043dcd1ba x1872053300072704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785329783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14710.206984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14718.255891] LustreError: 475074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14718.325385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14719.347963] LustreError: 475075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14719.404344] LustreError: 475075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14724.578578] LustreError: 499370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14724.590842] LustreError: 499370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14725.336988] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14726.663355] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14731.751653] LustreError: 499610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14731.801865] LustreError: 499610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14739.674503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14740.101543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14745.596061] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14746.176221] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14754.474542] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14760.016971] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14791.165174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14791.181370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14791.182325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14791.206743] Lustre: Skipped 1 previous similar message [14791.241944] Lustre: Skipped 1 previous similar message [14795.559888] Lustre: server umount lustre-OST0000 complete [14801.376278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14801.396996] LustreError: Skipped 1 previous similar message [14801.405576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14801.428199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14807.013388] LustreError: 499374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14807.046844] LustreError: 499374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14807.375953] Lustre: server umount lustre-MDT0000 complete [14813.116801] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785329892 with bad export cookie 14697965335437278272 [14813.125385] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14813.145860] LustreError: 473905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14814.273863] Lustre: server umount lustre-MDT0001 complete [14834.321855] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [14840.263443] Key type lgssc unregistered [14841.242218] LNet: 502188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14841.253409] LNetError: 502188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14841.273950] LNet: Removed LNI 192.168.201.108@tcp [14841.282245] LNet: Removed LNI 192.168.201.108@tcp10 [14841.292654] LNet: Removed LNI 192.168.201.108@tcp11 [14841.299491] LNet: Removed LNI 192.168.201.108@tcp12 [14841.309340] LNet: Removed LNI 192.168.201.108@tcp13 [14841.318246] LNet: Removed LNI 192.168.201.108@tcp14 [14841.349140] LNet: Removed LNI 192.168.201.108@tcp15 [14841.359990] LNet: Removed LNI 192.168.201.108@tcp16 [14841.364188] LNet: Removed LNI 192.168.201.108@tcp17 [14841.377822] LNet: Removed LNI 192.168.201.108@tcp18 [14841.382156] LNet: Removed LNI 192.168.201.108@tcp19 [14841.388440] LNet: Removed LNI 192.168.201.108@tcp20 [14841.396235] LNet: Removed LNI 192.168.201.108@tcp21 [14841.406515] LNet: Removed LNI 192.168.201.108@tcp22 [14841.421239] LNet: Removed LNI 192.168.201.108@tcp23 [14841.427196] LNet: Removed LNI 192.168.201.108@tcp24 [14841.438882] LNet: Removed LNI 192.168.201.108@tcp25 [14841.442181] LNet: Removed LNI 192.168.201.108@tcp26 [14841.447894] LNet: Removed LNI 192.168.201.108@tcp27 [14841.464344] LNet: Removed LNI 192.168.201.108@tcp28 [14841.482925] LNet: Removed LNI 192.168.201.108@tcp29 [14841.487402] LNet: Removed LNI 192.168.201.108@tcp30 [14841.493984] LNet: Removed LNI 192.168.201.108@tcp31 [14841.495789] LNet: Removed LNI 192.168.201.108@tcp32 [14841.518684] LNet: Removed LNI 192.168.201.108@tcp33 [14841.520568] LNet: Removed LNI 192.168.201.108@tcp34 [14841.531327] LNet: Removed LNI 192.168.201.108@tcp35 [14841.547955] LNet: Removed LNI 192.168.201.108@tcp36 [14841.552141] LNet: Removed LNI 192.168.201.108@tcp37 [14841.560259] LNet: Removed LNI 192.168.201.108@tcp38 [14841.567874] LNet: Removed LNI 192.168.201.108@tcp39 [14841.578943] LNet: Removed LNI 192.168.201.108@tcp40 [14841.595902] LNet: Removed LNI 192.168.201.108@tcp41 [14841.611712] LNet: Removed LNI 192.168.201.108@tcp42 [14841.613299] LNet: Removed LNI 192.168.201.108@tcp43 [14841.631304] LNet: Removed LNI 192.168.201.108@tcp44 [14841.638292] LNet: Removed LNI 192.168.201.108@tcp45 [14841.649263] LNet: Removed LNI 192.168.201.108@tcp46 [14841.651325] LNet: Removed LNI 192.168.201.108@tcp47 [14841.662051] LNet: Removed LNI 192.168.201.108@tcp48 [14841.666725] LNet: Removed LNI 192.168.201.108@tcp49 [14841.694377] LNet: Removed LNI 192.168.201.108@tcp50 [14842.991171] Key type .llcrypt unregistered [14842.993618] Key type ._llcrypt unregistered [14876.151871] Key type ._llcrypt registered [14876.189480] Key type .llcrypt registered [14876.495124] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [14904.558920] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [14906.709235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14906.867962] alg: No test for adler32 (adler32-zlib) [14907.918372] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [14908.143577] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [14909.875179] Key type lgssc registered [14911.267411] Lustre: Echo OBD driver; http://www.lustre.org/ [14963.938878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14963.988204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14965.389572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14965.447743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14965.723507] Lustre: lustre-MDT0000: new disk, initializing [14965.900922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14965.933521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14972.779829] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [14987.169253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14987.501290] Lustre: 505393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14987.684589] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14987.687637] Lustre: Skipped 1 previous similar message [14987.757879] Lustre: lustre-MDT0001: new disk, initializing [14987.836294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14987.899215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14987.917975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14995.258902] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15008.754422] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15015.668513] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15026.251304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15026.431782] Lustre: *** cfs_fail_loc=90f, val=0*** [15026.523987] Lustre: lustre-OST0000: new disk, initializing [15026.531821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15026.544552] Lustre: 507425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15026.691561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15032.395435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15032.417316] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15032.626031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15034.779306] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15045.883428] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15058.404883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15058.416918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15058.428734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15058.443549] Lustre: Skipped 1 previous similar message [15060.656830] Lustre: server umount lustre-OST0000 complete [15068.659913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15068.672696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15068.673127] Lustre: Skipped 2 previous similar messages [15068.697372] Lustre: Skipped 1 previous similar message [15072.381914] Lustre: server umount lustre-MDT0000 complete [15076.940519] LustreError: 505386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785330156 with bad export cookie 9647906947549900625 [15076.966734] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15077.625796] Lustre: server umount lustre-MDT0001 complete [15095.117351] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [15099.624168] Key type lgssc unregistered [15099.979535] LNet: 509332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15099.993346] LNetError: 509332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15100.011856] LNet: Removed LNI 192.168.201.108@tcp [15101.140757] Key type .llcrypt unregistered [15101.142748] Key type ._llcrypt unregistered [15124.144759] Key type ._llcrypt registered [15124.148537] Key type .llcrypt registered [15124.356648] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [15125.792787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15125.826463] alg: No test for adler32 (adler32-zlib) [15127.161995] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [15127.939679] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [15129.687214] Key type lgssc registered [15131.886031] Lustre: Echo OBD driver; http://www.lustre.org/ [15144.088830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15144.106063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15145.706096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15151.007326] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15156.195960] LustreError: 510602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15156.217757] LustreError: 510602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15160.030707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15160.520607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15165.014303] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15176.607767] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15186.228404] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15196.288686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15196.709335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15204.822595] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15213.521820] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15230.757668] loop0: detected capacity change from 0 to 204800000 [15231.065654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15231.088962] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15231.122320] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15240.980673] loop0: detected capacity change from 0 to 204800000 [15241.292186] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15241.306986] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15241.323068] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15249.244072] loop0: detected capacity change from 0 to 204800000 [15256.307168] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15256.502556] Lustre: fs15246-OST0000: new disk, initializing [15256.507849] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15256.512554] Lustre: 514199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15256.558770] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15265.339513] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15276.795295] loop1: detected capacity change from 0 to 204800000 [15283.980342] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15284.194918] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15284.219202] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15284.292483] Lustre: fs15246-MDT0000: new disk, initializing [15284.459352] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15284.488332] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15284.651932] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15285.664437] Lustre: 515324:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002322beaa x1872054407189120/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785330381 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15290.448616] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15294.823542] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15294.838910] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15294.900572] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15319.538728] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15319.560642] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15321.818946] Lustre: server umount fs15246-MDT0000 complete [15324.664779] LustreError: 511961:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15329.765874] LustreError: 510599:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15330.738485] Lustre: server umount fs15246-OST0000 complete [15344.099586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15344.101379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15344.106137] LustreError: Skipped 1 previous similar message [15344.107322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15344.141829] Lustre: Skipped 1 previous similar message [15345.124591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15345.131279] Lustre: Skipped 2 previous similar messages [15350.278880] Lustre: server umount lustre-OST0000 complete [15354.848099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15354.860143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15354.880142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15360.482517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15360.486466] Lustre: Skipped 2 previous similar messages [15368.159439] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15368.355904] Lustre: server umount lustre-MDT0000 complete [15369.699141] LustreError: 510598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15369.726919] LustreError: 510598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15372.633985] LustreError: 510582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785330452 with bad export cookie 4007721186726694706 [15372.654712] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15372.654846] LustreError: 510582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15372.961987] Lustre: server umount lustre-MDT0001 complete [15386.139900] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [15389.938926] Key type lgssc unregistered [15390.501921] LNet: 518424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15390.526637] LNetError: 518424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15390.556832] LNet: Removed LNI 192.168.201.108@tcp [15391.876206] Key type .llcrypt unregistered [15391.882410] Key type ._llcrypt unregistered [15412.807787] Key type ._llcrypt registered [15412.810166] Key type .llcrypt registered [15412.986065] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [15414.005452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15414.014524] alg: No test for adler32 (adler32-zlib) [15415.277309] Lustre: Lustre: Build Version: 2.17.56_1_g88845ec [15415.696317] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [15417.519144] Key type lgssc registered [15419.029203] Lustre: Echo OBD driver; http://www.lustre.org/ [15453.866632] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [15460.900916] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [15486.281478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15486.308804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15487.461427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15487.511290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15487.601862] Lustre: lustre-MDT0000: new disk, initializing [15487.674480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15487.692240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15491.292116] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [15499.294106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15499.367746] Lustre: 522888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15499.399162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15499.401069] Lustre: Skipped 1 previous similar message [15499.455669] Lustre: lustre-MDT0001: new disk, initializing [15499.516350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15499.548554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15499.563638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15504.087359] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all