[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499775273 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002189] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007429] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.012007] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.015057] *** VALIDATE selinux *** [ 0.023236] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027597] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032287] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035279] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041641] debug: unmapping init [mem 0xffffffffb1659000-0xffffffffb1660fff] [ 0.043198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044593] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.050983] ... event mask: 000000070000000f [ 0.051304] rcu: Hierarchical SRCU implementation. [ 0.053182] smp: Bringing up secondary CPUs ... [ 0.054568] x86: Booting SMP configuration: [ 0.055037] .... node #0, CPUs: #1 #2 #3 [ 0.058288] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147669] node 0 deferred pages initialised in 82ms [ 0.149110] devtmpfs: initialized [ 0.150181] x86/mm: Memory block size: 128MB [ 0.152766] gcov: version magic: 0x41383552 [ 0.153633] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158278] pinctrl core: initialized pinctrl subsystem [ 0.160134] [ 0.160472] ************************************************************* [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164009] ** ** [ 0.165008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167011] ** ** [ 0.168007] ** This means that this kernel is built to expose internal ** [ 0.171010] ** IOMMU data structures, which may compromise security on ** [ 0.172009] ** your system. ** [ 0.174010] ** ** [ 0.176009] ** If you see this message and you are not debugging the ** [ 0.177008] ** kernel, report this immediately to your vendor! ** [ 0.179007] ** ** [ 0.180006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182009] ************************************************************* [ 0.184539] NET: Registered protocol family 16 [ 0.185291] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.187042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192032] cpuidle: using governor menu [ 0.193527] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195347] PCI: Using configuration type 1 for base access [ 0.197092] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.203123] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.205031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209027] cryptd: max_cpu_qlen set to 1000 [ 0.211198] ACPI: Added _OSI(Module Device) [ 0.212014] ACPI: Added _OSI(Processor Device) [ 0.213043] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218805] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225270] ACPI: Interpreter enabled [ 0.226046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227009] ACPI: Using IOAPIC for interrupt routing [ 0.229112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232333] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239256] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.246086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250435] acpiphp: Slot [2] registered [ 0.252112] acpiphp: Slot [5] registered [ 0.253101] acpiphp: Slot [6] registered [ 0.254100] acpiphp: Slot [7] registered [ 0.256124] acpiphp: Slot [8] registered [ 0.257057] acpiphp: Slot [9] registered [ 0.258112] acpiphp: Slot [10] registered [ 0.259134] acpiphp: Slot [3] registered [ 0.260078] acpiphp: Slot [4] registered [ 0.261056] acpiphp: Slot [11] registered [ 0.262096] acpiphp: Slot [12] registered [ 0.264077] acpiphp: Slot [13] registered [ 0.265073] acpiphp: Slot [14] registered [ 0.266102] acpiphp: Slot [15] registered [ 0.267086] acpiphp: Slot [16] registered [ 0.269073] acpiphp: Slot [17] registered [ 0.270082] acpiphp: Slot [18] registered [ 0.271084] acpiphp: Slot [19] registered [ 0.272089] acpiphp: Slot [20] registered [ 0.274073] acpiphp: Slot [21] registered [ 0.275104] acpiphp: Slot [22] registered [ 0.276054] acpiphp: Slot [23] registered [ 0.277081] acpiphp: Slot [24] registered [ 0.278095] acpiphp: Slot [25] registered [ 0.279076] acpiphp: Slot [26] registered [ 0.280072] acpiphp: Slot [27] registered [ 0.282098] acpiphp: Slot [28] registered [ 0.283074] acpiphp: Slot [29] registered [ 0.284079] acpiphp: Slot [30] registered [ 0.285078] acpiphp: Slot [31] registered [ 0.286052] PCI host bridge to bus 0000:00 [ 0.287024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.297019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298128] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301019] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303413] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.324596] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327502] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332681] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335816] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343989] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.347014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351386] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.383020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.396425] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412064] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.430016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440550] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.450020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.458017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.480204] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.487015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.503019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.519015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.529000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.538017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.551017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.570017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.580906] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.587021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.593015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.608023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.618304] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.621394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.623318] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.626382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.628217] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.634105] iommu: Default domain type: Passthrough [ 0.636432] SCSI subsystem initialized [ 0.638153] ACPI: bus type USB registered [ 0.640092] usbcore: registered new interface driver usbfs [ 0.642104] usbcore: registered new interface driver hub [ 0.644081] usbcore: registered new device driver usb [ 0.646177] pps_core: LinuxPPS API ver. 1 registered [ 0.648010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.651057] PTP clock support registered [ 0.654058] EDAC MC: Ver: 3.0.0 [ 0.657168] PCI: Using ACPI for IRQ routing [ 0.658819] NetLabel: Initializing [ 0.660015] NetLabel: domain hash size = 128 [ 0.662011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.664125] NetLabel: unlabeled traffic allowed by default [ 0.667040] vgaarb: loaded [ 0.669256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.670011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.675337] clocksource: Switched to clocksource kvm-clock [ 0.789186] VFS: Disk quotas dquot_6.6.0 [ 0.790823] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.793580] *** VALIDATE ramfs *** [ 0.794561] *** VALIDATE hugetlbfs *** [ 0.796375] pnp: PnP ACPI init [ 0.798541] pnp: PnP ACPI: found 6 devices [ 0.816716] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.820048] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.822257] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.824608] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.827298] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830093] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.832947] NET: Registered protocol family 2 [ 0.835421] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840378] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844142] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.849446] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853149] TCP: Hash tables configured (established 65536 bind 65536) [ 0.856249] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.859430] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.862421] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.865713] NET: Registered protocol family 1 [ 0.868378] RPC: Registered named UNIX socket transport module. [ 0.870783] RPC: Registered udp transport module. [ 0.872581] RPC: Registered tcp transport module. [ 0.874474] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.877101] NET: Registered protocol family 44 [ 0.878868] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.881634] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.884090] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.886523] PCI: CLS 0 bytes, default 64 [ 0.888299] Unpacking initramfs... [ 2.267233] debug: unmapping init [mem 0xffff8e1c7cc54000-0xffff8e1c7ffbffff] [ 2.270616] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.273246] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.276638] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.768922] Initialise system trusted keyrings [ 2.770831] Key type blacklist registered [ 2.773032] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.784894] zbud: loaded [ 2.788419] *** VALIDATE nfs *** [ 2.789391] *** VALIDATE nfs4 *** [ 2.790588] pstore: using deflate compression [ 2.793518] Platform Keyring initialized [ 2.888570] NET: Registered protocol family 38 [ 2.890628] Key type asymmetric registered [ 2.891832] Asymmetric key parser 'x509' registered [ 2.893279] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.896246] io scheduler mq-deadline registered [ 2.897943] io scheduler kyber registered [ 2.899825] io scheduler bfq registered [ 2.901879] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.905484] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.907754] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.910934] ACPI: Power Button [PWRF] [ 2.916719] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.923516] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.932789] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.941758] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.959166] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.989155] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.020072] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.025336] Non-volatile memory driver v1.3 [ 3.027277] Linux agpgart interface v0.103 [ 3.058895] virtio_blk virtio1: [vda] 145920 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.061880] vda: detected capacity change from 0 to 74711040 [ 3.079523] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.082645] vdb: detected capacity change from 0 to 1073741824 [ 3.097442] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.100301] vdc: detected capacity change from 0 to 2621440000 [ 3.113252] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.115715] vdd: detected capacity change from 0 to 2621440000 [ 3.128611] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.131589] vde: detected capacity change from 0 to 4294967296 [ 3.145426] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.148036] vdf: detected capacity change from 0 to 4294967296 [ 3.154289] libphy: Fixed MDIO Bus: probed [ 3.179396] usbcore: registered new interface driver usbserial_generic [ 3.182236] usbserial: USB Serial support registered for generic [ 3.184874] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.188425] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.189866] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.192215] mousedev: PS/2 mouse device common for all mice [ 3.195665] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.197637] rtc_cmos 00:05: RTC can wake from S4 [ 3.202731] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.203548] rtc_cmos 00:05: registered as rtc0 [ 3.208494] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.210497] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.213657] intel_pstate: CPU model not supported [ 3.222599] hid: raw HID events driver (C) Jiri Kosina [ 3.225308] usbcore: registered new interface driver usbhid [ 3.227490] usbhid: USB HID core driver [ 3.229196] drop_monitor: Initializing network drop monitor service [ 3.231671] Initializing XFRM netlink socket [ 3.233764] NET: Registered protocol family 10 [ 3.237221] Segment Routing with IPv6 [ 3.238274] NET: Registered protocol family 17 [ 3.240024] mpls_gso: MPLS GSO support [ 3.244503] RAS: Correctable Errors collector initialized. [ 3.246196] AVX version of gcm_enc/dec engaged. [ 3.247825] AES CTR mode by8 optimization enabled [ 3.317609] sched_clock: Marking stable (3317584206, 0)->(4208404514, -890820308) [ 3.321210] registered taskstats version 1 [ 3.323060] Loading compiled-in X.509 certificates [ 3.324950] zswap: loaded using pool lzo/zbud [ 3.348695] Key type big_key registered [ 3.362458] Key type encrypted registered [ 3.363573] ima: No TPM chip found, activating TPM-bypass! [ 3.365460] ima: Allocated hash algorithm: sha1 [ 3.366662] ima: No architecture policies found [ 3.367950] evm: Initialising EVM extended attributes: [ 3.369323] evm: security.selinux [ 3.370448] evm: security.ima [ 3.371298] evm: security.capability [ 3.372284] evm: HMAC attrs: 0x1 [ 3.374301] rtc_cmos 00:05: setting system clock to 2026-08-14 22:23:25 UTC (1786746205) [ 3.378597] debug: unmapping init [mem 0xffffffffb2603000-0xffffffffb27fffff] [ 3.381255] debug: unmapping init [mem 0xffffffffb1382000-0xffffffffb1658fff] [ 3.389134] Write protecting the kernel read-only data: 28672k [ 3.392775] debug: unmapping init [mem 0xffffffffafa03000-0xffffffffafbfffff] [ 3.395930] debug: unmapping init [mem 0xffffffffb0314000-0xffffffffb03fffff] [ 3.435312] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.444217] systemd[1]: Detected virtualization kvm. [ 3.445511] systemd[1]: Detected architecture x86-64. [ 3.446669] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.470817] systemd[1]: No hostname configured. [ 3.472811] systemd[1]: Set hostname to . [ 3.475168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.477937] systemd[1]: Initializing machine ID from random generator. [ 3.527704] random: ln: uninitialized urandom read (6 bytes read) [ 3.620414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.623231] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.628438] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.633390] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.092256] device-mapper: uevent: version 1.0.3 [ 5.100951] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.989969] virtio_net virtio0 ens2: renamed from eth0 [ 7.138738] random: fast init done [ 7.315265] scsi host0: ata_piix [ 7.417681] scsi host1: ata_piix [ 7.423174] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.434893] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.896146] random: crng init done [ 12.898610] random: 7 urandom warning(s) missed due to ratelimiting [ 13.738555] dracut-initqueue[579]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.042106] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.467515] printk: systemd: 17 output lines suppressed due to ratelimiting [ 20.473717] SELinux: Disabled at runtime. [ 20.675327] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.727241] systemd[1]: Detected virtualization kvm. [ 20.734834] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.943484] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.960220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.003236] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.018726] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.026182] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.069070] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.097979] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ 23.209377] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 25.253488] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.325787] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.329798] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.649302] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.936444] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.652725] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 33.755336] NFS: Registering the id_resolver key type [ 33.771882] Key type id_resolver registered [ 33.775970] Key type id_legacy registered [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg237-server login: [ 47.251064] hrtimer: interrupt took 5056605 ns [ 100.030547] libcfs: loading out-of-tree module taints kernel. [ 100.118741] Key type ._llcrypt registered [ 100.121630] Key type .llcrypt registered [ 100.229373] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 121.600202] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 123.951361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.957530] alg: No test for adler32 (adler32-zlib) [ 125.821620] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 127.201569] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 129.080254] Key type lgssc registered [ 131.234624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.807847] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.836265] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 208.123924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.148606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.306863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.385737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.588930] Lustre: lustre-MDT0000: new disk, initializing [ 209.797117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.836603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 216.286520] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.483863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.624538] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.691467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.696158] Lustre: Skipped 1 previous similar message [ 230.816343] Lustre: lustre-MDT0001: new disk, initializing [ 230.915957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.983190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.996702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.273404] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.648567] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.740696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.089645] Lustre: lustre-OST0000: new disk, initializing [ 250.093558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.098535] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.159580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.695712] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.593865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.602679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.671559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 272.272206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.403663] Lustre: lustre-OST0001: new disk, initializing [ 272.407938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.413398] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.477047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.091201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 278.102539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 278.231575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.972257] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.612654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.316756] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.031364] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing check_logdir /tmp/testlogs/ [ 315.603324] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing yml_node [ 321.315826] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 324.245796] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 328.374751] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 330.931315] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 18:28:51 EDT 2026 [ 352.948608] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 354.572937] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 374.244267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 374.256179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.285277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.780410] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.784370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 375.797745] Lustre: Skipped 2 previous similar messages [ 375.816678] Lustre: Skipped 3 previous similar messages [ 377.939473] Lustre: server umount lustre-MDT0000 complete [ 381.812095] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746583 with bad export cookie 9915214886280103605 [ 381.815463] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 381.819338] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 386.017278] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 386.025666] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.049976] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 386.079973] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.137308] LustreError: 12159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 391.137518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 391.157501] LustreError: 12159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 391.163759] Lustre: Skipped 2 previous similar messages [ 392.165134] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.182662] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 396.256163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 396.482616] Lustre: server umount lustre-MDT0001 complete [ 400.469442] Lustre: server umount lustre-OST0000 complete [ 405.456567] Lustre: server umount lustre-OST0001 complete [ 412.393563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 421.159726] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 456.618540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.836829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 456.870137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 456.963809] Lustre: lustre-MDT0000: new disk, initializing [ 457.040281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.058662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 462.208946] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 473.027881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.182188] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 473.271812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 473.277082] Lustre: Skipped 1 previous similar message [ 473.397743] Lustre: lustre-MDT0001: new disk, initializing [ 473.558428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 473.605614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 473.625710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 478.062662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 486.615073] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 491.858861] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 498.971958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 499.407513] Lustre: lustre-OST0000: new disk, initializing [ 499.417540] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 499.430274] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.523388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.221116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 501.232080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 501.338779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 505.742278] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 515.525234] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.312668] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.650913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.801891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.116955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.941318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.953609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.957845] LustreError: Skipped 1 previous similar message [ 531.958556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.958560] Lustre: Skipped 2 previous similar messages [ 532.009355] Lustre: Skipped 2 previous similar messages [ 535.059566] Lustre: server umount lustre-OST0000 complete [ 540.141468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.150386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.164881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 540.173573] Lustre: Skipped 1 previous similar message [ 540.187652] Lustre: Skipped 2 previous similar messages [ 545.970981] Lustre: server umount lustre-MDT0000 complete [ 550.370605] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746752 with bad export cookie 9915214886280105131 [ 550.382249] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.388387] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.395713] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 551.092692] Lustre: server umount lustre-MDT0001 complete [ 555.611528] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:32:37 (1786746757) [ 558.068491] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 559.849427] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:32:41 (1786746761) [ 566.127682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.514029] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.617621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.371813] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 575.977587] LustreError: 19461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.023330] LustreError: 19461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 587.151196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.911268] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 604.417931] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 611.890463] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 619.224562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.308658] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 635.903504] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 643.553576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 643.565505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.582967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 643.588159] Lustre: Skipped 2 previous similar messages [ 649.182708] Lustre: server umount lustre-OST0000 complete [ 654.826867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 654.835646] LustreError: Skipped 1 previous similar message [ 659.435995] Lustre: server umount lustre-MDT0000 complete [ 661.986390] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.009244] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 664.203728] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786746866 with bad export cookie 9915214886280106020 [ 664.216582] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 664.784860] Lustre: server umount lustre-MDT0001 complete [ 676.931945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 679.659530] Key type lgssc unregistered [ 680.056538] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.063299] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.091902] LNet: Removed LNI 192.168.202.137@tcp [ 681.491213] Key type .llcrypt unregistered [ 681.495351] Key type ._llcrypt unregistered [ 706.186527] Key type ._llcrypt registered [ 706.189718] Key type .llcrypt registered [ 706.361356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 708.249411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 708.417737] alg: No test for adler32 (adler32-zlib) [ 709.749296] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 710.177169] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 711.937516] Key type lgssc registered [ 713.535100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 723.316680] loop: module loaded [ 723.467676] loop0: detected capacity change from 0 to 204800000 [ 723.891804] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.902906] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 724.390925] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.286615] loop0: detected capacity change from 0 to 204800000 [ 732.536288] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.576913] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 732.601805] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 738.941006] loop0: detected capacity change from 0 to 204800000 [ 745.019549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.050584] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.342854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 746.412293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 746.500512] Lustre: lustre-MDT0000: new disk, initializing [ 746.593284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.619392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 751.642501] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 761.429831] loop1: detected capacity change from 0 to 204800000 [ 767.163685] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 767.585511] Lustre: lustre-OST07c6: new disk, initializing [ 767.590864] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 767.598416] Lustre: Skipped 1 previous similar message [ 767.609553] Lustre: 26189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 767.751694] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 774.381316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 774.391820] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 774.435078] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 774.457893] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 779.447323] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 779.452509] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 779.456944] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 779.461470] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 779.466135] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 779.470327] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 799.566575] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 799.578046] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.604455] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.616321] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 799.623830] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 799.637372] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 804.644141] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 804.675182] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 804.688852] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 804.696766] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.717686] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 804.733329] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.748658] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 804.754865] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.766594] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 804.772392] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 804.783068] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 804.797371] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.100234] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 808.112496] Lustre: 25176:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 808.122483] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 808.135432] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.145676] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.161268] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.167288] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 808.175267] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.181642] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 808.190699] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 808.196501] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.205528] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.090194] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 813.104538] Lustre: 25174:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 813.114528] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 813.119416] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.133753] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 813.140864] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.148608] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 813.156465] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.160197] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 813.177761] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 813.194927] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 813.203524] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 821.485833] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 821.497879] Lustre: 25175:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 821.507637] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 821.517400] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.523437] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 821.532111] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.538274] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 821.545666] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.551555] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 821.557532] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 821.563184] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 821.569421] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 877.026722] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.036950] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 877.036993] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.984715] Lustre: server umount lustre-OST07c6 complete [ 894.808857] Lustre: server umount lustre-MDT0000 complete [ 916.832727] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 919.536732] Key type lgssc unregistered [ 920.048339] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.132407] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.192935] LNet: Removed LNI 192.168.202.137@tcp [ 922.380535] Key type .llcrypt unregistered [ 922.389782] Key type ._llcrypt unregistered [ 947.982549] Key type ._llcrypt registered [ 947.988768] Key type .llcrypt registered [ 948.215589] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 949.286516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.327280] alg: No test for adler32 (adler32-zlib) [ 950.558377] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 950.966316] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 952.738574] Key type lgssc registered [ 953.725556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.946757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 965.977811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.765764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.658974] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 977.898121] LustreError: 30025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.924773] LustreError: 30025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 983.023563] LustreError: 30026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.358821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.882235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 990.321619] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 999.804231] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1007.358613] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1016.110187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.438565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.795253] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1035.365940] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1038.013096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1058.505818] Lustre: server umount lustre-OST0000 complete [ 1063.906132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.908322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1063.926068] Lustre: Skipped 1 previous similar message [ 1063.943745] Lustre: Skipped 1 previous similar message [ 1069.031337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.054955] Lustre: Skipped 1 previous similar message [ 1073.416320] Lustre: server umount lustre-MDT0000 complete [ 1074.149714] Lustre: 29336:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747260/real 1786747260] req@00000000237166a8 x1873540182739712/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786747276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1074.159139] LustreError: 30026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.211495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.232494] LustreError: 30026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1076.193374] Lustre: 29335:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786747262/real 1786747262] req@00000000e2860c20 x1873540182740096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786747278 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1077.140160] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747279 with bad export cookie 8651412824546589252 [ 1077.151907] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.155246] LustreError: 30005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1077.412776] Lustre: server umount lustre-MDT0001 complete [ 1092.318794] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1095.890266] Key type lgssc unregistered [ 1096.192806] LNet: 33768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.201392] LNetError: 33768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.217720] LNet: Removed LNI 192.168.202.137@tcp [ 1097.140533] Key type .llcrypt unregistered [ 1097.142276] Key type ._llcrypt unregistered [ 1115.621754] Key type ._llcrypt registered [ 1115.623926] Key type .llcrypt registered [ 1115.756143] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1117.040923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1117.166541] alg: No test for adler32 (adler32-zlib) [ 1118.328736] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1118.579855] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1120.400365] Key type lgssc registered [ 1121.970172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1131.918324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1131.933713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.638648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1139.869316] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1143.777336] LustreError: 35044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.793384] LustreError: 35044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1148.904608] LustreError: 35045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.666907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.643344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1157.807338] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1167.094890] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1174.697127] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1183.515614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1183.893812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.649238] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1201.641569] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1203.412712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1219.052306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.056367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1219.073377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1223.913073] Lustre: server umount lustre-OST0000 complete [ 1229.282744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.290626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.315575] Lustre: Skipped 2 previous similar messages [ 1229.338677] Lustre: Skipped 2 previous similar messages [ 1234.404527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.413069] Lustre: Skipped 1 previous similar message [ 1234.975765] Lustre: server umount lustre-MDT0000 complete [ 1239.492431] LustreError: 35024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747441 with bad export cookie 4814602665280492393 [ 1239.510554] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.186563] Lustre: server umount lustre-MDT0001 complete [ 1252.453866] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1255.552417] Key type lgssc unregistered [ 1255.827722] LNet: 38602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.848026] LNetError: 38602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.873465] LNet: Removed LNI 192.168.202.137@tcp [ 1258.035352] Key type .llcrypt unregistered [ 1258.037418] Key type ._llcrypt unregistered [ 1286.246228] Key type ._llcrypt registered [ 1286.247503] Key type .llcrypt registered [ 1286.342515] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1287.660774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.820146] alg: No test for adler32 (adler32-zlib) [ 1288.949763] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1289.246130] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1291.089079] Key type lgssc registered [ 1292.893165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1302.414927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1302.427241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.127845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.304416] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1314.277349] LustreError: 40018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1314.312988] LustreError: 40018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1319.242899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.404678] LustreError: 40019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.729280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1325.159244] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1334.855449] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1341.928534] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1350.399903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.903105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.221824] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1367.026549] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1374.544903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1384.054473] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1384.941021] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.948501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.967899] Lustre: Skipped 2 previous similar messages [ 1384.980312] Lustre: Skipped 2 previous similar messages [ 1386.167915] Lustre: server umount lustre-MDT0000 complete [ 1389.138071] LustreError: 42107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.169892] LustreError: 40000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747592 with bad export cookie 5935302799967302358 [ 1390.184242] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1390.192611] LustreError: 40000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1394.254406] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.270555] LustreError: 40013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1394.279987] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [ 1395.170888] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.856729] Lustre: server umount lustre-MDT0001 complete [ 1418.955710] Lustre: server umount lustre-OST0000 complete [ 1437.457340] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1440.752436] Key type lgssc unregistered [ 1441.048609] LNet: 43907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.060772] LNetError: 43907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.075311] LNet: Removed LNI 192.168.202.137@tcp [ 1442.227898] Key type .llcrypt unregistered [ 1442.234680] Key type ._llcrypt unregistered [ 1470.972052] Key type ._llcrypt registered [ 1470.976706] Key type .llcrypt registered [ 1471.104266] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1472.029688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.068503] alg: No test for adler32 (adler32-zlib) [ 1473.194947] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1473.471996] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1475.220087] Key type lgssc registered [ 1476.646676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1484.964779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1484.995224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.553351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.318274] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1496.035353] LustreError: 45326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.058076] LustreError: 45326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1501.163632] LustreError: 45327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.909814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1503.211582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1507.845464] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1515.955466] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1522.914620] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1531.307306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1531.724869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.736854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1540.738461] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1551.443858] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1558.130886] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1558.506684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.532260] Lustre: Skipped 1 previous similar message [ 1558.546827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1563.219487] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1563.232282] Lustre: Skipped 2 previous similar messages [ 1564.044747] Lustre: server umount lustre-OST0000 complete [ 1584.104709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.128447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.145961] Lustre: Skipped 2 previous similar messages [ 1588.259087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.314990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.319751] Lustre: Skipped 2 previous similar messages [ 1596.384216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1596.613311] Lustre: server umount lustre-MDT0000 complete [ 1598.439869] LustreError: 45327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.469095] LustreError: 45327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1601.148066] LustreError: 45308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786747803 with bad export cookie 850854843824002423 [ 1601.149763] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.160183] LustreError: 45308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1607.725145] Lustre: server umount lustre-MDT0001 complete [ 1621.687965] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1625.042377] Key type lgssc unregistered [ 1625.315544] LNet: 49043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.323515] LNetError: 49043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.332804] LNet: Removed LNI 192.168.202.137@tcp [ 1626.485331] Key type .llcrypt unregistered [ 1626.492450] Key type ._llcrypt unregistered [ 1652.510651] Key type ._llcrypt registered [ 1652.512502] Key type .llcrypt registered [ 1652.626517] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1654.024362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1654.191956] alg: No test for adler32 (adler32-zlib) [ 1655.334802] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1655.631843] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1657.288470] Key type lgssc registered [ 1658.646671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1666.354274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1666.374654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.892264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.129494] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1677.289917] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.326833] LustreError: 50453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1682.407252] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.514463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.918797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1687.502911] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1696.352621] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1702.292231] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1709.401031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1709.760843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.904688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1716.271140] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1723.934624] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1733.330351] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1736.874523] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1748.948491] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1418h52m30s [ 1750.683599] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1753.062089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.068525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.076060] Lustre: Skipped 2 previous similar messages [ 1753.080433] Lustre: Skipped 1 previous similar message [ 1754.148389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.157881] Lustre: Skipped 2 previous similar messages [ 1757.291464] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 1757.304835] Lustre: Skipped 1 previous similar message [ 1758.723594] Lustre: server umount lustre-MDT0000 complete [ 1760.230258] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.250801] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1762.511343] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1764.430870] LustreError: 50449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.859957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.970526] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.191102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.255380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1770.235749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1774.469704] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1778.094649] Lustre: DEBUG MARKER: Wait for df (36801) ... [ 1780.112164] Lustre: DEBUG MARKER: done [ 1784.801392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1784.806492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.818418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.822637] Lustre: Skipped 1 previous similar message [ 1785.837111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.981422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1790.985049] Lustre: Skipped 2 previous similar messages [ 1799.137080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1799.295597] Lustre: server umount lustre-OST0000 complete [ 1806.314109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.325377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.325823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1806.340019] Lustre: Skipped 1 previous similar message [ 1806.349378] Lustre: Skipped 2 previous similar messages [ 1809.554340] Lustre: server umount lustre-MDT0000 complete [ 1813.478392] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1813.492925] LustreError: 50454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1813.519901] LustreError: 50434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748015 with bad export cookie 1659747208771352552 [ 1813.525507] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.529200] LustreError: 50434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1813.857954] Lustre: server umount lustre-MDT0001 complete [ 1824.194496] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 1827.163943] Key type lgssc unregistered [ 1827.416085] LNet: 55384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.424640] LNetError: 55384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.448907] LNet: Removed LNI 192.168.202.137@tcp [ 1828.422773] Key type .llcrypt unregistered [ 1828.424920] Key type ._llcrypt unregistered [ 1869.998920] Key type ._llcrypt registered [ 1870.001435] Key type .llcrypt registered [ 1870.141086] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 1871.197291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1871.222153] alg: No test for adler32 (adler32-zlib) [ 1872.346720] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 1872.560107] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 1874.320480] Key type lgssc registered [ 1875.603579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.691646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1884.736317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.160716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1886.176609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1886.351791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.840352] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1902.991218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.055866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1903.078930] Lustre: 57607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1903.259675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.718309] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1921.662372] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1928.239633] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1935.775688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.908818] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1936.152642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.670571] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 1941.685405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1949.560137] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1957.117258] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1418h55m58s [ 1958.696635] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1966.184501] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1968.207578] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1968.271764] LustreError: 57612:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1988.772191] Lustre: lustre-MDT0000: Client d559dbcb-cf1d-492a-ae52-e872da62169d (at 192.168.202.37@tcp) reconnecting [ 2008.352209] LustreError: 57612:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2010.731968] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2013.766453] Lustre: DEBUG MARKER: done [ 2023.393614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.409085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.422494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.907609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.924225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.943706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.467465] Lustre: server umount lustre-OST0000 complete [ 2031.607501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.619551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.650836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.812475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.834103] Lustre: Skipped 4 previous similar messages [ 2042.339432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.363687] Lustre: Skipped 1 previous similar message [ 2045.408385] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2045.637945] Lustre: server umount lustre-MDT0000 complete [ 2050.251372] LustreError: 57599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748252 with bad export cookie 6875149929412480259 [ 2050.267394] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.271432] LustreError: 57599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2050.752674] Lustre: server umount lustre-MDT0001 complete [ 2063.075321] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2066.549811] Key type lgssc unregistered [ 2066.970898] LNet: 61939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.977384] LNetError: 61939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.017529] LNet: Removed LNI 192.168.202.137@tcp [ 2069.226911] Key type .llcrypt unregistered [ 2069.230057] Key type ._llcrypt unregistered [ 2119.609907] Key type ._llcrypt registered [ 2119.611959] Key type .llcrypt registered [ 2119.755736] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2121.083762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.205502] alg: No test for adler32 (adler32-zlib) [ 2122.306843] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2122.561094] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2124.248176] Key type lgssc registered [ 2125.354558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2128.743608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2128.775284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.137426] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2130.174849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.393985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.679794] Lustre: Failing over lustre-MDT0000 [ 2132.262170] Lustre: server umount lustre-MDT0000 complete [ 2139.745865] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 18:59:01 (1786748341) [ 2147.208447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.864180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.182563] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2162.554867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.645487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2162.707062] Lustre: 64739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2163.056577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2168.647137] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2177.875459] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2184.585548] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2191.716709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2191.946216] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2192.376881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.252976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2199.394972] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2208.363250] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2215.904315] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2224.610159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2224.617848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.639711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.198070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.201392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.246234] Lustre: Skipped 1 previous similar message [ 2229.890836] Lustre: server umount lustre-MDT0000 complete [ 2233.927225] LustreError: 64730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748436 with bad export cookie 10446947541865537778 [ 2233.937036] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.410757] Lustre: server umount lustre-MDT0001 complete [ 2238.459155] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2243.316157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2252.695788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.816450] Lustre: 63662:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748440/real 1786748440] req@00000000361ef1b7 x1873541410630016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748456 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.837301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.915205] Lustre: 63661:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748445/real 1786748445] req@000000008973955b x1873541410630144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2259.607389] LustreError: 66589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.635772] LustreError: 66589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2259.801883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.956093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2260.708222] Lustre: 63660:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748446/real 1786748446] req@0000000057471415 x1873541410630400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2262.817509] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.602954] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2264.878255] LustreError: 66589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.891065] LustreError: 66589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2266.080477] Lustre: 63663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748451/real 1786748451] req@00000000f441d159 x1873541410630528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2270.176569] Lustre: 63663:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748456/real 1786748456] req@0000000045638224 x1873541410630656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2270.184616] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.257333] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2273.421769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.621492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.290716] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2278.897392] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2283.058522] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2287.015960] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2294.767927] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.830423] Lustre: server umount lustre-MDT0000 complete [ 2303.475118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.476431] LustreError: 69084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.490667] Lustre: Skipped 2 previous similar messages [ 2303.525047] LustreError: 69084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2305.278583] LustreError: 64730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748507 with bad export cookie 10446947541865540053 [ 2305.284339] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2305.290060] LustreError: 64730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2305.591581] Lustre: server umount lustre-MDT0001 complete [ 2312.954703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2322.946961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.003031] Lustre: 63660:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786748510/real 1786748510] req@000000000a40222e x1873541410657664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786748526 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2324.030360] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.510851] LustreError: 66589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.583481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.641977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2334.862921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2342.890865] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2343.016323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2348.174945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2348.533166] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2356.398434] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.033681] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2374.114926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.119213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.125208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2374.125220] Lustre: Skipped 2 previous similar messages [ 2374.156770] Lustre: Skipped 2 previous similar messages [ 2375.430607] Lustre: server umount lustre-MDT0000 complete [ 2379.245770] LustreError: 64729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748581 with bad export cookie 10446947541865541432 [ 2379.247365] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.254840] LustreError: 64729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2379.263845] LustreError: 71207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.280268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2379.280576] LustreError: 71207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2379.296613] Lustre: Skipped 2 previous similar messages [ 2379.561677] Lustre: server umount lustre-MDT0001 complete [ 2386.899740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2390.517485] Lustre: DEBUG MARKER: files compared the same [ 2392.704135] Lustre: server umount lustre-OST0000 complete [ 2411.322621] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2414.509675] Key type lgssc unregistered [ 2414.946638] LNet: 74482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.952835] LNetError: 74482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.971378] LNet: Removed LNI 192.168.202.137@tcp [ 2416.207366] Key type .llcrypt unregistered [ 2416.209812] Key type ._llcrypt unregistered [ 2441.303550] Key type ._llcrypt registered [ 2441.308590] Key type .llcrypt registered [ 2441.434290] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2442.887606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2442.932400] alg: No test for adler32 (adler32-zlib) [ 2444.119351] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2444.397850] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2446.104200] Key type lgssc registered [ 2446.917745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.459649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2454.479525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.036174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.559549] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2466.273086] LustreError: 75908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.296043] LustreError: 75908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2469.573349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.929542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.405809] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2483.077303] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2491.105346] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2499.666688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2499.875988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2500.924102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2507.376230] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug +malloc all [ 2517.664674] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2524.641463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2524.647846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.657277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2524.660103] Lustre: Skipped 1 previous similar message [ 2524.669752] Lustre: Skipped 1 previous similar message [ 2529.763804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2529.772844] Lustre: Skipped 1 previous similar message [ 2531.004947] Lustre: server umount lustre-OST0000 complete [ 2536.422028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2536.428582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.439269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.017258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.019784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.577779] Lustre: server umount lustre-MDT0000 complete [ 2545.121614] LustreError: 75907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.142657] LustreError: 75907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2545.763189] LustreError: 75887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748747 with bad export cookie 7961164947319070845 [ 2545.771035] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.781857] LustreError: 75887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2546.104284] Lustre: server umount lustre-MDT0001 complete [ 2557.841868] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2561.286666] Key type lgssc unregistered [ 2561.729536] LNet: 79444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2561.750468] LNetError: 79444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2561.797335] LNet: Removed LNI 192.168.202.137@tcp [ 2563.130621] Key type .llcrypt unregistered [ 2563.134308] Key type ._llcrypt unregistered [ 2587.100864] Key type ._llcrypt registered [ 2587.102747] Key type .llcrypt registered [ 2587.225350] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2588.436217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2588.623302] alg: No test for adler32 (adler32-zlib) [ 2589.938948] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2590.208796] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2591.945714] Key type lgssc registered [ 2593.273750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2600.934793] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2600.958562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.482160] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2621.600347] LustreError: 80835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [ 2648.288798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.096555] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2664.095633] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2672.228468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.328518] Lustre: *** cfs_fail_loc=706, val=0*** [ 2672.900638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2673.007161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2676.803657] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2678.244939] LustreError: 80860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.266976] LustreError: 80860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2683.388357] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.415873] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2685.249153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.657563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.816504] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2698.944468] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2706.083307] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2711.529140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.532966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2711.542992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2715.734497] Lustre: server umount lustre-OST0000 complete [ 2721.762196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.764284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.783584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2721.803343] Lustre: Skipped 2 previous similar messages [ 2721.824327] Lustre: Skipped 2 previous similar messages [ 2726.841891] Lustre: server umount lustre-MDT0000 complete [ 2726.895655] LustreError: 81924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.504646] LustreError: 80843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786748932 with bad export cookie 11871655662927417916 [ 2730.514229] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2730.516115] LustreError: 80843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2730.889589] Lustre: server umount lustre-MDT0001 complete [ 2742.078516] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 2745.000478] Key type lgssc unregistered [ 2745.250770] LNet: 84453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2745.257194] LNetError: 84453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2746.282109] LNet: Removed LNI 192.168.202.137@tcp [ 2747.269474] Key type .llcrypt unregistered [ 2747.277240] Key type ._llcrypt unregistered [ 2772.837141] Key type ._llcrypt registered [ 2772.840179] Key type .llcrypt registered [ 2773.025451] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2774.345476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2774.533047] alg: No test for adler32 (adler32-zlib) [ 2775.601199] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 2775.794088] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 2777.456195] Key type lgssc registered [ 2778.851233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.133966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.158344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.402304] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2801.492258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.524608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2807.779239] LustreError: 86556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2807.811632] LustreError: 86556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2810.877179] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2811.901323] LustreError: 86559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.953995] LustreError: 86559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2816.997302] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.006766] LustreError: 86560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2822.115509] LustreError: 86555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.126811] LustreError: 86555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2822.778756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.095380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.185997] LustreError: 86559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.215253] LustreError: 86559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2830.961842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2837.475871] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.483118] LustreError: 86554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2841.202096] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.212111] Lustre: MGS: Client 3732b5c7-f383-4096-b0e6-1605ba8224e3 (at 0@lo) reconnecting [ 2841.225495] LustreError: 86570:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2841.539208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.649040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2846.850368] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2863.073767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2863.083625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2863.116929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.181730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2867.190897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2867.196556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.387976] Lustre: server umount lustre-OST0000 complete [ 2872.298461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.299199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2872.337817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.358361] Lustre: Skipped 2 previous similar messages [ 2877.411046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.413771] Lustre: Skipped 1 previous similar message [ 2878.180062] Lustre: server umount lustre-MDT0000 complete [ 2882.436503] Lustre: server umount lustre-MDT0001 complete [ 2886.437784] Lustre: server umount MGS complete [ 2897.389612] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:11:39 (1786749099) [ 2932.865407] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 2947.018270] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 2984.965104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.965232] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 2999.780544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.968527] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3000.010622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3000.086474] Lustre: lustre-MDT0001: new disk, initializing [ 3000.168302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.193549] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3000.205546] Lustre: 93751:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3000.216361] LustreError: 93751:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3004.089603] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3015.138301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.528581] Lustre: lustre-OST0000: new disk, initializing [ 3015.533119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3015.541490] Lustre: 94516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3015.640224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.521910] Lustre: 94815:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005c595dc4 x1873542095349120/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786749241 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3023.523910] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3034.746732] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3041.832843] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.846098] Lustre: MGS: Client 48338388-727b-42b1-9571-3d4a19c2220d (at 0@lo) reconnecting [ 3041.876571] LustreError: 93737:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3041.909460] Lustre: 93020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3041.939805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3042.058756] Lustre: lustre-MDT0000: new disk, initializing [ 3042.208779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.272731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3042.335877] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3043.374738] Lustre: 95586:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000045638224 x1873542095356288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786749261 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3047.532938] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3048.554846] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3052.533536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3052.542632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3052.629507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3077.611972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.619632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.625133] Lustre: Skipped 2 previous similar messages [ 3077.651655] Lustre: Skipped 1 previous similar message [ 3083.111345] Lustre: server umount lustre-OST0000 complete [ 3087.844937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3087.847673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.858758] Lustre: Skipped 1 previous similar message [ 3087.859226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3087.897397] Lustre: Skipped 3 previous similar messages [ 3092.499714] Lustre: server umount lustre-MDT0000 complete [ 3092.972107] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.990921] LustreError: 96010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3103.208223] Lustre: server umount lustre-MDT0001 complete [ 3107.285430] Lustre: server umount MGS complete [ 3120.238389] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:15:22 (1786749322) [ 3143.060441] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3146.042124] Key type lgssc unregistered [ 3146.445897] LNet: 98086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3146.451163] LNetError: 98086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3146.462423] LNet: Removed LNI 192.168.202.137@tcp [ 3147.399441] Key type .llcrypt unregistered [ 3147.403769] Key type ._llcrypt unregistered [ 3164.341432] Key type ._llcrypt registered [ 3164.349787] Key type .llcrypt registered [ 3164.477800] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3165.785161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3166.024512] alg: No test for adler32 (adler32-zlib) [ 3167.233662] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 3167.568572] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3169.312904] Key type lgssc registered [ 3170.694975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3179.688249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3179.731277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.184430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.253173] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3191.778545] LustreError: 99310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.815308] LustreError: 99310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3196.082168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.656323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.458136] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3209.830468] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3216.116892] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3223.679464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.217166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.327332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3230.617927] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3239.548994] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3252.202318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.204725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3252.228569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3252.253204] Lustre: Skipped 1 previous similar message [ 3253.838511] Lustre: server umount lustre-OST0000 complete [ 3258.337892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3258.351650] LustreError: Skipped 1 previous similar message [ 3258.360265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.378922] Lustre: Skipped 1 previous similar message [ 3258.384545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.438191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3262.445702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.556558] Lustre: server umount lustre-MDT0000 complete [ 3267.555777] LustreError: 99306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.576352] LustreError: 99306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3268.580626] LustreError: 99290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749470 with bad export cookie 7722129109726255684 [ 3268.594052] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.600947] LustreError: 99290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3269.175642] Lustre: server umount lustre-MDT0001 complete [ 3281.821664] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3284.507072] Key type lgssc unregistered [ 3284.827879] LNet: 102845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3284.837388] LNetError: 102845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3285.861883] LNet: Removed LNI 192.168.202.137@tcp [ 3286.635490] Key type .llcrypt unregistered [ 3286.637627] Key type ._llcrypt unregistered [ 3301.738694] Key type ._llcrypt registered [ 3301.742503] Key type .llcrypt registered [ 3301.846142] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3303.202870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.296511] alg: No test for adler32 (adler32-zlib) [ 3304.375863] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 3304.565735] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3306.304274] Key type lgssc registered [ 3307.725177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.270359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3314.278673] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3314.306812] LustreError: 103923:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3314.314742] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3314.324233] LustreError: 103918:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3314.328834] LustreError: 103918:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3314.339955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.879485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.234592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.490714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.328708] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3334.409559] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lsmod [ 3337.316780] LustreError: 105127:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3337.322912] LustreError: 105128:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3337.329171] LustreError: 105127:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 3337.334149] LustreError: 105127:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3337.339864] LustreError: 105127:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3337.361771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3337.761312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3340.836731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3340.881933] LustreError: 103980:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3343.842211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3343.848191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.858146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.893262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.896200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.900401] Lustre: Skipped 1 previous similar message [ 3345.908575] Lustre: Skipped 1 previous similar message [ 3347.245621] Lustre: server umount lustre-MDT0000 complete [ 3348.014246] LustreError: 103961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.039789] LustreError: 103961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3351.572164] LustreError: 103940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749553 with bad export cookie 5577714745760666846 [ 3351.579439] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.581159] LustreError: 103940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3352.120289] LustreError: 104391:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000051a111c2 x1873542650025088/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3352.282150] Lustre: server umount lustre-MDT0001 complete [ 3366.822223] Lustre: server umount lustre-OST0000 complete [ 3375.982309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.433151] LustreError: 106192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3376.526792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.016993] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3381.732948] LustreError: 106193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.849795] LustreError: 106192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.975919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.447723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3395.680623] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3404.091382] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3410.331681] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3417.733740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.130224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.240887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3424.310627] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3432.917381] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3445.731605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.734140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.742874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.752964] Lustre: Skipped 1 previous similar message [ 3445.765444] Lustre: Skipped 1 previous similar message [ 3450.850372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3450.868321] Lustre: Skipped 1 previous similar message [ 3451.670486] Lustre: server umount lustre-OST0000 complete [ 3457.001395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3457.012160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.034874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3461.093077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.693369] Lustre: server umount lustre-MDT0000 complete [ 3466.211975] LustreError: 106193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.223965] LustreError: 106193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3466.696911] LustreError: 106173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749668 with bad export cookie 5577714745760667791 [ 3466.709949] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3466.710894] LustreError: 106173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3467.030729] Lustre: server umount lustre-MDT0001 complete [ 3481.636772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3485.429074] Key type lgssc unregistered [ 3485.811737] LNet: 109746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3485.824820] LNetError: 109746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.891823] LNet: Removed LNI 192.168.202.137@tcp [ 3488.546511] Key type .llcrypt unregistered [ 3488.552876] Key type ._llcrypt unregistered [ 3514.893433] Key type ._llcrypt registered [ 3514.895965] Key type .llcrypt registered [ 3515.067331] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3516.284840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3516.414427] alg: No test for adler32 (adler32-zlib) [ 3517.565346] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 3517.819406] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3519.523520] Key type lgssc registered [ 3520.398651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3528.682740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3528.740241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.270312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.582912] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3540.464761] LustreError: 111162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3540.477987] LustreError: 111162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3545.519670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.572051] LustreError: 111163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.822807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3550.760448] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3557.999469] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3564.439295] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3572.431873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3572.932381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3575.073545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3578.991702] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3587.527586] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.067693] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3598.906501] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3605.986508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3606.011782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.041725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3610.942673] Lustre: server umount lustre-OST0000 complete [ 3616.227782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.251061] Lustre: Skipped 1 previous similar message [ 3616.256829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.260850] Lustre: Skipped 2 previous similar messages [ 3621.348143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.355026] Lustre: Skipped 1 previous similar message [ 3621.869801] Lustre: server umount lustre-MDT0000 complete [ 3625.707077] LustreError: 111898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749827 with bad export cookie 10579459500973215225 [ 3625.723340] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.729347] LustreError: 111898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3626.084807] Lustre: server umount lustre-MDT0001 complete [ 3638.480403] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3641.423783] Key type lgssc unregistered [ 3641.756210] LNet: 114804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3641.762972] LNetError: 114804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3642.791474] LNet: Removed LNI 192.168.202.137@tcp [ 3643.721878] Key type .llcrypt unregistered [ 3643.724292] Key type ._llcrypt unregistered [ 3660.805537] Key type ._llcrypt registered [ 3660.807333] Key type .llcrypt registered [ 3660.919769] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3661.918732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.009053] alg: No test for adler32 (adler32-zlib) [ 3663.137320] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 3663.450898] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3665.088250] Key type lgssc registered [ 3666.290418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3674.733887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3674.748086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.192214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.938680] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3686.377790] LustreError: 116029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.417347] LustreError: 116029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3691.008904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.377153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3696.780608] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3707.602897] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3717.330454] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3726.371916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3727.099812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.052834] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3744.910542] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3746.551880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3762.153921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.158809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.183547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3762.206132] Lustre: Skipped 1 previous similar message [ 3766.939914] Lustre: server umount lustre-OST0000 complete [ 3772.389702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.419914] Lustre: Skipped 1 previous similar message [ 3772.432444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.505810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.517335] Lustre: Skipped 2 previous similar messages [ 3778.253562] Lustre: server umount lustre-MDT0000 complete [ 3782.625898] LustreError: 116030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3783.390517] LustreError: 116010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786749985 with bad export cookie 5805970449995125386 [ 3783.397611] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3783.404364] LustreError: 116010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3783.944066] Lustre: server umount lustre-MDT0001 complete [ 3798.104638] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3801.497751] Key type lgssc unregistered [ 3801.712474] LNet: 119590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3801.720190] LNetError: 119590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3801.731464] LNet: Removed LNI 192.168.202.137@tcp [ 3802.694309] Key type .llcrypt unregistered [ 3802.696453] Key type ._llcrypt unregistered [ 3830.361668] Key type ._llcrypt registered [ 3830.365427] Key type .llcrypt registered [ 3830.515857] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 3832.139303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3832.411605] alg: No test for adler32 (adler32-zlib) [ 3833.727431] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 3834.125321] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 3835.920179] Key type lgssc registered [ 3837.414660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3846.864544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3846.892318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.242308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3853.347369] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3858.405024] LustreError: 121008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.421548] LustreError: 121008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3863.529055] LustreError: 121008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3864.249298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.843444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3871.091317] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3879.615745] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3887.159083] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.690972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.249132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.585872] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 3912.593429] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3918.833460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3924.345886] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3928.544348] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3928.558882] Lustre: Skipped 2 previous similar messages [ 3931.243705] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3931.294571] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3933.230625] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3933.241306] Lustre: Skipped 2 previous similar messages [ 3936.620562] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3936.626439] Lustre: Skipped 1 previous similar message [ 3944.507914] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3944.520705] Lustre: Skipped 2 previous similar messages [ 3945.787844] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.37@tcp 192.168.0.[2,10]@tcp [ 3951.145693] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.37/24@tcp in log lustre-MDT0000 [ 3951.153622] Lustre: Skipped 2 previous similar messages [ 3954.277311] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 3954.287136] Lustre: Skipped 1 previous similar message [ 3960.802732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3960.817054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.834915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3961.831374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3961.835472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.843521] LustreError: Skipped 2 previous similar messages [ 3961.873681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3962.789605] Lustre: server umount lustre-OST0000 complete [ 3967.460644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3967.467622] LustreError: Skipped 2 previous similar messages [ 3967.472080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3967.489776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.028489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.033845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.060479] Lustre: Skipped 1 previous similar message [ 3973.454582] Lustre: server umount lustre-MDT0000 complete [ 3975.138115] LustreError: 123498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.161043] LustreError: 123498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3977.829192] LustreError: 120988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750179 with bad export cookie 5934757601970765396 [ 3977.843168] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3977.847245] LustreError: 120988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3978.200628] Lustre: server umount lustre-MDT0001 complete [ 3991.726235] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 3995.317953] Key type lgssc unregistered [ 3995.767723] LNet: 125811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3995.773172] LNetError: 125811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3995.793066] LNet: Removed LNI 192.168.202.137@tcp [ 3996.729832] Key type .llcrypt unregistered [ 3996.734279] Key type ._llcrypt unregistered [ 4033.485324] Key type ._llcrypt registered [ 4033.492692] Key type .llcrypt registered [ 4033.671552] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4035.116173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.158138] alg: No test for adler32 (adler32-zlib) [ 4036.481867] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 4036.860442] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4038.696190] Key type lgssc registered [ 4040.223382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4051.521865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4051.571526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4053.402540] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4053.442301] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.37/24@tcp [ 4053.464278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.432530] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4063.721426] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.738311] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4068.853314] LustreError: 127376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.181354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.626793] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4070.667321] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.37/24@tcp [ 4070.733979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4076.240636] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4086.500223] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4094.377684] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4102.870548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4103.316768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.681347] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4118.419837] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.833503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4134.374647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4134.376069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.378829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4134.424368] Lustre: Skipped 1 previous similar message [ 4135.931784] Lustre: server umount lustre-OST0000 complete [ 4142.562990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4142.572892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.580603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4142.583107] Lustre: Skipped 1 previous similar message [ 4144.614169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.616747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.804560] Lustre: server umount lustre-MDT0000 complete [ 4149.736341] LustreError: 127376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.744407] LustreError: 127376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4150.632809] LustreError: 127357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750352 with bad export cookie 4992869432854513470 [ 4150.641126] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.645341] LustreError: 127357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4151.191615] Lustre: server umount lustre-MDT0001 complete [ 4161.954747] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4164.939961] Key type lgssc unregistered [ 4165.360679] LNet: 130969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4165.372550] LNetError: 130969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4165.402608] LNet: Removed LNI 192.168.202.137@tcp [ 4166.617352] Key type .llcrypt unregistered [ 4166.619069] Key type ._llcrypt unregistered [ 4197.325876] Key type ._llcrypt registered [ 4197.333511] Key type .llcrypt registered [ 4197.503770] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4212.330449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4213.720139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4213.890266] alg: No test for adler32 (adler32-zlib) [ 4215.062706] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 4215.301634] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4216.995395] Key type lgssc registered [ 4218.844461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4259.336491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4259.375824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.790715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4260.817830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4260.909320] Lustre: lustre-MDT0000: new disk, initializing [ 4260.978935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.992830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4265.883876] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4279.001630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.140612] Lustre: 134379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4279.203203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4279.208876] Lustre: Skipped 1 previous similar message [ 4279.277650] Lustre: lustre-MDT0001: new disk, initializing [ 4279.368423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.407688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4279.418719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4284.391421] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4295.809334] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4302.802853] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4309.940149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4310.235467] Lustre: lustre-OST0000: new disk, initializing [ 4310.240448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4310.252704] Lustre: 136350:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4310.343804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.333853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4317.357095] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4317.574568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4318.721603] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4329.708678] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.971520] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4334.140536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4338.646719] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4338.962115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4343.284427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.289275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.294722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4343.303539] Lustre: Skipped 1 previous similar message [ 4343.342225] LustreError: Skipped 1 previous similar message [ 4347.821572] Lustre: server umount lustre-OST0000 complete [ 4353.505632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.512281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4353.527621] Lustre: Skipped 1 previous similar message [ 4353.545384] Lustre: Skipped 2 previous similar messages [ 4358.626551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.642875] Lustre: Skipped 1 previous similar message [ 4358.991947] Lustre: server umount lustre-MDT0000 complete [ 4362.784783] LustreError: 136349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750564 with bad export cookie 12832941794572488782 [ 4362.788974] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.793103] LustreError: 136349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4363.113793] Lustre: server umount lustre-MDT0001 complete [ 4373.826296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.347111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4379.638583] LustreError: 138568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.692593] LustreError: 138568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4380.199668] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4384.743818] LustreError: 138569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.865953] LustreError: 138568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.051903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.565303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4395.956326] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4406.363433] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4411.980711] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.082694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4419.504929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.460233] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4435.013537] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4440.543779] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4440.792486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.354292] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4445.491699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4453.672943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.881518] Lustre: lustre-OST0001: new disk, initializing [ 4453.883881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4453.900093] Lustre: 141938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4454.041481] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4459.207159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4459.224198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4459.414860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4462.011541] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4471.326985] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4471.569502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4476.728543] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4477.011927] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4484.333924] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff897809b41800.ost_server_uuid 50 [ 4485.867471] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff897809b41800.ost_server_uuid in IDLE state after 0 sec [ 4500.461255] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4500.463979] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4500.473367] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4500.499980] Lustre: Skipped 1 previous similar message [ 4505.578164] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4505.597110] Lustre: Skipped 2 previous similar messages [ 4510.176828] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4510.337126] Lustre: server umount lustre-OST0001 complete [ 4510.701554] LustreError: 140389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.812439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.814442] LustreError: 142161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.824763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.836833] Lustre: Skipped 1 previous similar message [ 4515.872630] LustreError: 142161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4520.605540] Lustre: server umount lustre-OST0000 complete [ 4526.051522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.863381] Lustre: server umount lustre-MDT0000 complete [ 4531.195904] LustreError: 138563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.239748] LustreError: 138563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4535.280323] LustreError: 143735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786750737 with bad export cookie 12832941794572489664 [ 4535.282236] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.286159] LustreError: 143735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.607645] Lustre: server umount lustre-MDT0001 complete [ 4556.600820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4560.318353] Key type lgssc unregistered [ 4560.578656] LNet: 144949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4560.582320] LNetError: 144949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4560.592412] LNet: Removed LNI 192.168.202.137@tcp [ 4561.446168] Key type .llcrypt unregistered [ 4561.447578] Key type ._llcrypt unregistered [ 4606.572213] Key type ._llcrypt registered [ 4606.574905] Key type .llcrypt registered [ 4606.679113] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4626.084690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4627.620643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4627.985037] alg: No test for adler32 (adler32-zlib) [ 4629.181789] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 4629.525957] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4631.264197] Key type lgssc registered [ 4632.744890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4670.870458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4670.933418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.278972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4672.328747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4672.457417] Lustre: lustre-MDT0000: new disk, initializing [ 4672.559225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.578435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4677.662603] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4689.491816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.638334] Lustre: 149107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4689.694852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4689.699276] Lustre: Skipped 1 previous similar message [ 4689.773442] Lustre: lustre-MDT0001: new disk, initializing [ 4689.863392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.893073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4689.905847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4694.652404] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4704.881068] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.804903] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4716.524657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4716.878662] Lustre: lustre-OST0000: new disk, initializing [ 4716.886397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4716.904258] Lustre: 151073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4716.984593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4722.888509] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4725.379719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4725.397246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4725.491330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4733.480681] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4738.965645] Lustre: Failing over lustre-OST0000 [ 4739.174291] Lustre: server umount lustre-OST0000 complete [ 4757.473908] Lustre: 147330:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750943/real 1786750943] req@000000006a09ac5b x1873544039400192/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786750959 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4757.476989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.506271] Lustre: 147330:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4757.536367] Lustre: Skipped 1 previous similar message [ 4757.784529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4758.009290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.027834] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4762.592183] Lustre: 147331:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750948/real 1786750948] req@0000000070c4526b x1873544039402112/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786750964 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4762.611782] Lustre: 147331:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4763.811945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4767.151111] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4767.178197] Lustre: 147331:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750953/real 1786750953] req@00000000159d6faa x1873544039403008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786750969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4767.212460] Lustre: 147331:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4767.799325] Lustre: Failing over lustre-MDT0000 [ 4767.811749] LustreError: 147328:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4767.868160] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 4767.880593] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4767.882149] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4768.058617] Lustre: server umount lustre-MDT0000 complete [ 4771.816689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4771.834227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4771.984124] Lustre: 147331:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750958/real 1786750958] req@00000000150994ec x1873544039404032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786750974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4777.441341] LustreError: 149118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.462795] LustreError: 149118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4778.060259] LustreError: 149112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.570665] LustreError: 149113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.583559] LustreError: 149113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4787.689511] LustreError: 149118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4787.721345] LustreError: 149118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4788.718632] Lustre: 147330:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786750974/real 1786750974] req@00000000f7946d9e x1873544039412864/t0(0) o400->MGC192.168.202.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786750990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4788.755713] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.910455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.826170] LustreError: 149113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.857854] LustreError: 149113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4797.927060] LustreError: 147328:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fa045005 x1873544039418112/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4798.319766] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 4798.374608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4798.444957] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4799.577464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4803.570118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4803.624564] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4803.699492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4804.062161] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4813.806913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.825753] Lustre: Skipped 2 previous similar messages [ 4813.835934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4818.927781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4818.948243] Lustre: Skipped 1 previous similar message [ 4819.585952] Lustre: server umount lustre-OST0000 complete [ 4829.153675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.155887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4829.177388] Lustre: Skipped 2 previous similar messages [ 4829.177727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4829.191036] Lustre: Skipped 1 previous similar message [ 4830.963687] Lustre: server umount lustre-MDT0000 complete [ 4834.275801] LustreError: 149117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.291741] LustreError: 149117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4834.579880] LustreError: 151084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751036 with bad export cookie 16326868613913731786 [ 4834.587276] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4834.590723] LustreError: 151084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4834.913809] Lustre: server umount lustre-MDT0001 complete [ 4848.349562] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 4851.761156] Key type lgssc unregistered [ 4852.205880] LNet: 154735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4852.214873] LNetError: 154735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4852.237963] LNet: Removed LNI 192.168.202.137@tcp [ 4853.292933] Key type .llcrypt unregistered [ 4853.295405] Key type ._llcrypt unregistered [ 4892.750520] Key type ._llcrypt registered [ 4892.753312] Key type .llcrypt registered [ 4892.937851] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 4894.328637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4894.473267] alg: No test for adler32 (adler32-zlib) [ 4895.875116] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 4896.346681] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 4898.232150] Key type lgssc registered [ 4900.218441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4911.111944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4911.136195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.745426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.619383] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 4922.353855] LustreError: 156256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.371296] LustreError: 156256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4927.457380] LustreError: 156257:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.750824] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4932.586993] LustreError: 156256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.731960] LustreError: 156285:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4934.748770] LustreError: 156285:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4934.763203] LustreError: 156285:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4935.504513] Lustre: server umount lustre-MDT0000 complete [ 4952.561538] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 4964.265788] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5001.509723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.809237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.844261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.959157] Lustre: lustre-MDT0000: new disk, initializing [ 5002.098995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.150234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5006.294161] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5016.627275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.788107] Lustre: 159589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5016.841237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5016.846213] Lustre: Skipped 1 previous similar message [ 5016.960272] Lustre: lustre-MDT0001: new disk, initializing [ 5017.033052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.058565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5017.063727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5022.982723] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5033.030111] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5038.680191] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5046.489748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5046.817931] Lustre: lustre-OST0000: new disk, initializing [ 5046.826610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5046.835122] Lustre: 161558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5046.966641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5053.010982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5053.021296] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5053.218812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.972851] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5065.367547] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5069.526777] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5071.755383] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:47:53 (1786751273) [ 5329.377981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5329.396700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.409996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5333.892942] Lustre: server umount lustre-MDT0000 complete [ 5335.011044] LustreError: 160721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5335.047874] LustreError: 160721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5337.807985] LustreError: 159580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751539 with bad export cookie 5930319226791200144 [ 5337.808389] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5337.820526] LustreError: 159580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5340.128661] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.130056] LustreError: 159596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.136166] Lustre: Skipped 2 previous similar messages [ 5340.144234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5340.178032] Lustre: Skipped 2 previous similar messages [ 5342.177283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5344.816856] Lustre: server umount lustre-MDT0001 complete [ 5357.396851] Lustre: server umount lustre-OST0000 complete [ 5368.333191] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5377.452937] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5416.272285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5416.613672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5416.652491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5416.716158] Lustre: lustre-MDT0000: new disk, initializing [ 5416.801708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5416.814448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5421.531182] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5432.194398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5432.318816] Lustre: 166224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5432.370332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5432.375034] Lustre: Skipped 1 previous similar message [ 5432.451444] Lustre: lustre-MDT0001: new disk, initializing [ 5432.610669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5432.645393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5432.668592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5437.736971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5447.715792] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.169442] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5460.242634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5460.464447] Lustre: lustre-OST0000: new disk, initializing [ 5460.466706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5460.471746] Lustre: 168190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5460.548961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5466.992916] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5469.282501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5469.294377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5469.377644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5475.549724] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5480.395282] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5480.643833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5485.364198] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5485.578101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5490.150018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.153342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5490.176916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5490.183654] LustreError: Skipped 1 previous similar message [ 5490.198507] Lustre: Skipped 1 previous similar message [ 5493.866793] Lustre: server umount lustre-OST0000 complete [ 5499.361444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5499.375641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.396444] Lustre: Skipped 1 previous similar message [ 5499.405910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.465724] Lustre: server umount lustre-MDT0000 complete [ 5505.508155] LustreError: 166234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.529945] LustreError: 166234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5508.270154] LustreError: 168200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751710 with bad export cookie 5930319226791358974 [ 5508.275727] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5508.294281] LustreError: 168200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5508.974298] Lustre: server umount lustre-MDT0001 complete [ 5518.413661] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:55:20 (1786751720) [ 5536.298569] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5546.493632] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5588.369187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.589619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5588.619544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5588.746609] Lustre: lustre-MDT0000: new disk, initializing [ 5588.850190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.865575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5594.565776] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5606.290342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.414966] Lustre: 172951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5606.741167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5611.238953] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5621.028585] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5628.750691] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5636.282751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5636.655593] Lustre: lustre-OST0000: new disk, initializing [ 5636.660893] Lustre: Skipped 1 previous similar message [ 5636.681615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5636.683512] Lustre: Skipped 2 previous similar messages [ 5636.704982] Lustre: 174927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5644.264131] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5644.910820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5644.918191] Lustre: Skipped 1 previous similar message [ 5644.932509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5645.119512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5654.616705] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5665.766922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5665.768192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.781405] LustreError: Skipped 1 previous similar message [ 5665.785565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5665.785579] Lustre: Skipped 2 previous similar messages [ 5665.834763] Lustre: Skipped 2 previous similar messages [ 5668.833782] Lustre: server umount lustre-OST0000 complete [ 5673.451510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5676.004988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.023291] Lustre: Skipped 1 previous similar message [ 5686.241964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.245298] Lustre: Skipped 7 previous similar messages [ 5687.264982] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5689.581533] Lustre: server umount lustre-MDT0000 complete [ 5691.360808] LustreError: 174426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.370346] LustreError: 174426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5693.794853] LustreError: 172941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751895 with bad export cookie 5930319226791359856 [ 5693.796575] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.816308] LustreError: 172941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5694.155642] Lustre: server umount lustre-MDT0001 complete [ 5701.913207] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:58:23 (1786751903) [ 5723.103743] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 5733.909486] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5775.801168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.088488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5776.111071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5776.179320] Lustre: lustre-MDT0000: new disk, initializing [ 5776.253919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.262513] Lustre: Skipped 2 previous similar messages [ 5776.273590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5780.665677] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5793.407110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.571833] Lustre: 179251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5793.923860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5799.967769] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5811.167497] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5817.109718] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5824.947710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5825.382693] Lustre: 181231:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5832.610524] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5834.340195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5834.475464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5842.761048] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.202358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.219797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.245829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.247155] Lustre: Skipped 1 previous similar message [ 5855.270278] Lustre: Skipped 1 previous similar message [ 5857.152882] Lustre: server umount lustre-OST0000 complete [ 5869.595159] Lustre: server umount lustre-MDT0000 complete [ 5870.565870] LustreError: 179265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.597432] LustreError: 179265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5873.630247] LustreError: 179244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752075 with bad export cookie 5930319226791360941 [ 5873.640421] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.641817] LustreError: 179244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5874.187441] Lustre: server umount lustre-MDT0001 complete [ 5886.164968] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 5889.126841] Key type lgssc unregistered [ 5889.450333] LNet: 183101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5889.471524] LNetError: 183101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5889.485699] LNet: Removed LNI 192.168.202.137@tcp [ 5890.491986] Key type .llcrypt unregistered [ 5890.495138] Key type ._llcrypt unregistered [ 5916.599101] Key type ._llcrypt registered [ 5916.601466] Key type .llcrypt registered [ 5916.691193] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 5918.037502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5918.169070] alg: No test for adler32 (adler32-zlib) [ 5919.163403] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 5919.428883] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 5921.088685] Key type lgssc registered [ 5922.551369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5931.343204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5931.367969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.124154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5937.902827] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5943.271169] LustreError: 184525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.313300] LustreError: 184525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5947.423913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.229991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5954.490826] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5962.697863] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5968.510307] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5976.862643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5977.511442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.596294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5985.290971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 5994.985704] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6014.446212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6014.467141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6014.477901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.533305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6018.541768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6018.542873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6018.553282] Lustre: Skipped 1 previous similar message [ 6020.785827] Lustre: server umount lustre-OST0000 complete [ 6028.786656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.807637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.400432] Lustre: server umount lustre-MDT0000 complete [ 6033.894063] LustreError: 185467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.902259] LustreError: 185467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6036.983583] LustreError: 184507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752239 with bad export cookie 5952204828596703325 [ 6036.986584] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6036.995295] LustreError: 184507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.464889] Lustre: server umount lustre-MDT0001 complete [ 6050.529772] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 6053.558765] Key type lgssc unregistered [ 6053.863745] LNet: 188091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6053.886379] LNetError: 188091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6053.914818] LNet: Removed LNI 192.168.202.137@tcp [ 6055.122992] Key type .llcrypt unregistered [ 6055.125270] Key type ._llcrypt unregistered [ 6082.777337] Key type ._llcrypt registered [ 6082.780658] Key type .llcrypt registered [ 6082.926640] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6084.072213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6084.130726] alg: No test for adler32 (adler32-zlib) [ 6085.283273] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 6085.618289] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 6087.296274] Key type lgssc registered [ 6088.683377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6100.485242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6100.536629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6102.259709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.826931] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6112.745353] LustreError: 189508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6112.785487] LustreError: 189508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6117.867344] LustreError: 189527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.508180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.032154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6124.751582] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6131.974909] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.706877] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6146.801491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.058475] LustreError: 191330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.378732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.702705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6154.034814] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6161.756651] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6173.153278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.162723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.188889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.836216] Lustre: lustre-OST0000: Not available for connect from 192.168.202.37@tcp (stopping) [ 6174.183531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6174.197763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6177.291550] Lustre: server umount lustre-OST0000 complete [ 6185.710776] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89781800a800.ost_server_uuid 50 [ 6188.069768] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6204.902927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.908043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6204.935126] Lustre: Skipped 1 previous similar message [ 6204.942614] Lustre: Skipped 3 previous similar messages [ 6205.344993] Lustre: server umount lustre-MDT0000 complete [ 6209.329396] LustreError: 189487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752411 with bad export cookie 5234250400994985696 [ 6209.334669] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6209.345509] LustreError: 189487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6209.794821] Lustre: server umount lustre-MDT0001 complete [ 6220.051835] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:07:01 (1786752421) [ 6226.963181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.431365] LustreError: 193423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.506455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.555873] LustreError: 193424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.189290] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6242.650030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.793948] LustreError: 193423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.824156] LustreError: 193423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6243.136322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6248.953341] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6257.418942] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6263.673745] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6272.097610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6272.411125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.674509] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6290.832913] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.014992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6298.360748] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.535935] Lustre: lustre-OST0001: new disk, initializing [ 6298.539175] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6298.545346] Lustre: 196266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6298.652824] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6304.336948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6304.344051] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6304.443043] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6305.788586] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6316.853996] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6322.661044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6322.683621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.704825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6324.706413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6326.482064] Lustre: server umount lustre-OST0000 complete [ 6327.886760] LustreError: 195248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.122813] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6334.350701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6339.114477] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6339.379678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6355.939683] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6355.948695] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.957877] LustreError: Skipped 1 previous similar message [ 6355.958871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6355.958876] Lustre: Skipped 4 previous similar messages [ 6356.018107] Lustre: Skipped 2 previous similar messages [ 6361.001203] Lustre: server umount lustre-OST0001 complete [ 6366.181720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.184166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6366.218689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.218699] Lustre: Skipped 1 previous similar message [ 6372.309402] Lustre: server umount lustre-MDT0000 complete [ 6376.426622] LustreError: 195685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.445400] LustreError: 195685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6376.573487] LustreError: 193404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752578 with bad export cookie 5234250400994986767 [ 6376.582101] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.583767] LustreError: 193404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6376.868780] Lustre: server umount lustre-MDT0001 complete [ 6403.627275] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:10:05 (1786752605) [ 6410.835507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.254362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6411.291517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6411.495638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6416.554921] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6425.373928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.478551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6425.510473] Lustre: 199705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6430.678648] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6439.508604] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.225355] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.645295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.920283] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6460.906597] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6462.587341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6468.322924] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.849094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.983058] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6477.247714] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6477.255832] Lustre: Skipped 2 previous similar messages [ 6484.710244] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6485.730765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6492.971370] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6496.235277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.261718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6496.262802] Lustre: Skipped 2 previous similar messages [ 6496.278035] Lustre: Skipped 4 previous similar messages [ 6510.049508] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6510.245588] Lustre: server umount lustre-OST0000 complete [ 6511.611470] LustreError: 201559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.656884] LustreError: 201559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6532.065903] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6532.067965] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.090420] LustreError: Skipped 1 previous similar message [ 6532.091545] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6532.091549] Lustre: Skipped 7 previous similar messages [ 6532.137407] Lustre: Skipped 1 previous similar message [ 6533.786215] Lustre: server umount lustre-OST0001 complete [ 6544.686794] Lustre: server umount lustre-MDT0000 complete [ 6549.366466] LustreError: 199697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752751 with bad export cookie 5234250400994988174 [ 6549.377801] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.383495] LustreError: 199697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6550.125542] Lustre: server umount lustre-MDT0001 complete [ 6577.563738] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:12:58 (1786752778) [ 6597.703954] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 6606.324785] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 6647.984613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6648.385278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6648.465365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6648.608995] Lustre: lustre-MDT0000: new disk, initializing [ 6648.679801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.699810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6653.167795] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6665.206592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6665.338992] Lustre: 207812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6665.386902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6665.390588] Lustre: Skipped 1 previous similar message [ 6665.461828] Lustre: lustre-MDT0001: new disk, initializing [ 6665.582971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6665.605484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6672.013029] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6684.461735] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6692.233778] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6699.590848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.891383] Lustre: lustre-OST0000: new disk, initializing [ 6699.898830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6699.906881] Lustre: 209781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6707.427640] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6710.365147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6710.379441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6710.509501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6718.436570] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6723.678721] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6723.975495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6728.723675] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6728.886394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6736.360085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6736.370091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.379433] Lustre: Skipped 2 previous similar messages [ 6736.390362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.404349] Lustre: Skipped 4 previous similar messages [ 6737.736745] Lustre: server umount lustre-OST0000 complete [ 6747.964916] Lustre: server umount lustre-MDT0000 complete [ 6751.713917] LustreError: 207823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.724865] LustreError: 207823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6752.776763] LustreError: 207803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752954 with bad export cookie 5234250400994989966 [ 6752.782846] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6752.783284] LustreError: 207803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6753.017423] Lustre: server umount lustre-MDT0001 complete [ 6763.101079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.727210] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6782.227702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6789.176602] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6799.275982] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6807.050489] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6816.016844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.583820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6833.484128] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6838.712672] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6838.929654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6843.484174] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6843.671197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6848.488352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6848.503233] LustreError: Skipped 1 previous similar message [ 6851.924853] Lustre: server umount lustre-OST0000 complete [ 6860.177835] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6865.555752] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6897.718523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6899.172655] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6899.216774] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6906.438089] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6914.595244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6914.612103] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6915.180923] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6916.966194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6924.777970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.783843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6924.793418] Lustre: Skipped 6 previous similar messages [ 6924.796626] Lustre: Skipped 7 previous similar messages [ 6927.439181] Lustre: server umount lustre-OST0000 complete [ 6938.900111] Lustre: server umount lustre-MDT0000 complete [ 6944.114344] LustreError: 211982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753146 with bad export cookie 5234250400994990848 [ 6944.141867] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6944.844191] Lustre: server umount lustre-MDT0001 complete [ 6954.721227] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:19:16 (1786753156) [ 6961.960120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.419432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.425472] Lustre: Skipped 6 previous similar messages [ 6967.796444] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6978.024868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.179936] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 6993.255037] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7001.082245] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7008.260413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.496193] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7023.259359] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7028.692699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7033.322534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7039.026609] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7039.240786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.977722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7047.242476] Lustre: lustre-OST0001: new disk, initializing [ 7047.246205] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7047.265466] Lustre: 221374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7055.468120] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7056.143345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7056.156215] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7056.330275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7068.800905] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7074.396909] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7074.693390] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7080.768332] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7081.250090] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7087.086923] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.091508] LustreError: Skipped 3 previous similar messages [ 7089.920208] Lustre: server umount lustre-OST0001 complete [ 7092.233326] LustreError: 219796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.273093] LustreError: 219796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7097.610994] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7098.089552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7103.890785] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7104.180739] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7134.930790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7140.351561] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7140.363578] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7140.369537] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7140.387733] Lustre: Skipped 1 previous similar message [ 7142.043414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7151.314886] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7153.263783] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7162.485948] Lustre: server umount lustre-OST0001 complete [ 7181.284854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.307516] Lustre: Skipped 6 previous similar messages [ 7181.311617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7181.313988] Lustre: Skipped 7 previous similar messages [ 7195.104136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7195.207241] Lustre: server umount lustre-OST0000 complete [ 7205.208099] Lustre: server umount lustre-MDT0000 complete [ 7209.144582] LustreError: 221373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753411 with bad export cookie 5234250400994991807 [ 7209.149489] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7209.157740] LustreError: 221373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7215.643844] Lustre: server umount lustre-MDT0001 complete [ 7240.267863] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:24:01 (1786753441) [ 7247.690471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.115341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7248.177694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7253.675298] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7263.254057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.329548] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7263.356760] Lustre: 226620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7268.773898] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7279.445058] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7287.017921] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7296.823688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.099675] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7304.303534] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7313.136468] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.026813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.148942] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7330.425665] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7339.064379] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7340.796851] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7346.024615] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7346.297173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7352.150226] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7352.420028] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7358.547221] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89780869d000.ost_server_uuid 50 [ 7360.557679] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89780869d000.ost_server_uuid in IDLE state after 0 sec [ 7361.986894] Lustre: Permanently deactivating lustre-OST0001 [ 7361.995214] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7366.278602] Lustre: Permanently reactivating lustre-OST0001 [ 7366.298204] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7366.301215] Lustre: Skipped 2 previous similar messages [ 7371.244289] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7371.256324] LustreError: Skipped 2 previous similar messages [ 7377.415083] Lustre: server umount lustre-OST0001 complete [ 7396.322388] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7396.635338] Lustre: server umount lustre-OST0000 complete [ 7406.982627] Lustre: server umount lustre-MDT0000 complete [ 7411.799362] LustreError: 229492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753613 with bad export cookie 5234250400994993102 [ 7411.814911] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7411.818878] LustreError: 229492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7412.255256] Lustre: server umount lustre-MDT0001 complete [ 7441.035680] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:27:22 (1786753642) [ 7450.337468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.691296] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7450.742337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7450.753209] Lustre: Skipped 2 previous similar messages [ 7456.621697] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7465.916598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.003132] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7466.065108] Lustre: 233318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.109553] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7480.047351] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7487.783105] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7496.911403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.263991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7497.310410] Lustre: Permanently deactivating lustre-OST0000 [ 7497.333350] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7497.619153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7497.629695] Lustre: Skipped 10 previous similar messages [ 7504.929067] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7506.728223] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7516.474958] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.963752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.059000] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7531.873791] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7532.406448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7540.375493] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7542.361135] Lustre: Permanently reactivating lustre-OST0000 [ 7542.380966] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7542.386592] Lustre: Skipped 2 previous similar messages [ 7548.020955] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7548.023684] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 00000000873a46c2, cur 1786753750 deadline 1786753737 last 1786753707 [ 7548.044845] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7548.073975] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7548.176433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7559.416969] Lustre: server umount lustre-OST0001 complete [ 7578.080593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7578.326320] Lustre: server umount lustre-OST0000 complete [ 7588.487142] Lustre: server umount lustre-MDT0000 complete [ 7592.314693] LustreError: 233310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753794 with bad export cookie 5234250400994994775 [ 7592.318917] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7592.324424] LustreError: 233310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7592.750390] Lustre: server umount lustre-MDT0001 complete [ 7603.940479] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 7607.038792] Key type lgssc unregistered [ 7607.386733] LNet: 238186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7607.390530] LNetError: 238186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7607.404895] LNet: Removed LNI 192.168.202.137@tcp [ 7608.324660] Key type .llcrypt unregistered [ 7608.331193] Key type ._llcrypt unregistered [ 7632.480278] Key type ._llcrypt registered [ 7632.482801] Key type .llcrypt registered [ 7632.646026] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7634.104834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7634.141803] alg: No test for adler32 (adler32-zlib) [ 7635.335785] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 7635.685295] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 7637.392168] Key type lgssc registered [ 7638.970863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7650.450167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7650.475582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.035672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7656.144230] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7662.576559] LustreError: 239655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.592867] LustreError: 239655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7665.550751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7665.752682] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7665.799329] Lustre: 239641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7665.817737] Lustre: Permanently deactivating lustre-MDT0001 [ 7665.835955] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7666.328432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7668.557749] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7668.565549] LustreError: 239685:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7668.584939] LustreError: 239685:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7668.611083] LustreError: 239685:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7670.996375] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7680.452596] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7687.755157] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7696.011772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7696.502871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.516400] LustreError: 241497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.552620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7703.034911] LustreError: 241569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7704.046842] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7708.131660] LustreError: 241499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.246414] LustreError: 241498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7712.288258] LustreError: 241498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7712.289588] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7713.945333] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7722.469306] LustreError: 242117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.516110] LustreError: 242117:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7723.597550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7723.801940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7729.157670] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7729.159311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7730.157948] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7739.444392] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7744.356951] Lustre: Permanently reactivating lustre-MDT0001 [ 7744.376864] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7744.380877] Lustre: Skipped 2 previous similar messages [ 7746.238788] Lustre: lustre-MDT0001: Client bfa24cae-2d18-409b-88a3-d84897ba1540 (at 192.168.202.37@tcp) reconnecting [ 7754.259739] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7755.978569] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7762.569381] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7762.845153] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7768.038282] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7768.259154] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7770.560717] Lustre: Permanently deactivating lustre-MDT0001 [ 7770.575226] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7770.579904] Lustre: Skipped 2 previous similar messages [ 7777.936028] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7789.027741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7789.041918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.062734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.564575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7790.571927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7790.599591] Lustre: Skipped 1 previous similar message [ 7790.633066] Lustre: Skipped 3 previous similar messages [ 7794.670902] Lustre: server umount lustre-MDT0000 complete [ 7795.686523] LustreError: 241016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.734033] LustreError: 241016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7799.122733] LustreError: 239635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754001 with bad export cookie 11421113577175010990 [ 7799.142798] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.750408] Lustre: server umount lustre-MDT0001 complete [ 7806.629180] Lustre: server umount lustre-OST0000 complete [ 7820.657458] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:33:42 (1786754022) [ 7833.088872] Lustre: server umount lustre-OST0001 complete [ 7841.001526] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 7850.205894] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 7892.106780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7892.502583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7892.510958] Lustre: Skipped 2 previous similar messages [ 7892.537183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7892.664743] Lustre: lustre-MDT0000: new disk, initializing [ 7892.761921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7892.794921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7897.813235] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7909.405387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.514857] Lustre: 247939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7909.601074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7909.605115] Lustre: Skipped 1 previous similar message [ 7909.703746] Lustre: lustre-MDT0001: new disk, initializing [ 7909.948794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.991046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7909.998913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7916.615296] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7929.173368] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7935.727175] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7944.010816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7944.564644] Lustre: lustre-OST0000: new disk, initializing [ 7944.571838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7944.606693] Lustre: 249911:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7944.805559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7950.492462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7950.522116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7950.644046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7954.446860] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7966.687543] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7968.564608] LustreError: 247945:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7977.653850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.768046] Lustre: lustre-OST0001: new disk, initializing [ 7977.770743] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7977.775844] Lustre: 251025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.846727] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7978.593704] LustreError: 247945:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7978.615530] LustreError: 249360:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7983.846959] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 7984.790600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7984.816522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7985.110476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7988.656391] LustreError: 249360:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7995.125900] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7996.999901] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8000.489747] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8000.503367] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.528249] Lustre: Skipped 1 previous similar message [ 8000.540592] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8000.554890] Lustre: Skipped 1 previous similar message [ 8005.598145] Lustre: server umount lustre-OST0001 complete [ 8005.622846] LustreError: 249897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.660713] LustreError: 249897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8026.086064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.100101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8026.103357] Lustre: Skipped 2 previous similar messages [ 8026.115145] Lustre: Skipped 1 previous similar message [ 8028.850863] Lustre: server umount lustre-OST0000 complete [ 8036.328116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8036.339935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8036.347910] Lustre: Skipped 1 previous similar message [ 8039.700856] Lustre: server umount lustre-MDT0000 complete [ 8043.651888] LustreError: 248735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754245 with bad export cookie 11421113577175013811 [ 8043.657743] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.664183] LustreError: 248735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8050.174826] Lustre: server umount lustre-MDT0001 complete [ 8060.994391] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8064.154929] Key type lgssc unregistered [ 8064.576457] LNet: 253058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8064.588122] LNetError: 253058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8064.607887] LNet: Removed LNI 192.168.202.137@tcp [ 8065.636326] Key type .llcrypt unregistered [ 8065.644267] Key type ._llcrypt unregistered [ 8111.051669] Key type ._llcrypt registered [ 8111.053322] Key type .llcrypt registered [ 8111.203070] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8113.080422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8113.100951] alg: No test for adler32 (adler32-zlib) [ 8114.323643] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8114.701828] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8116.400196] Key type lgssc registered [ 8117.812936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8127.632809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8127.678323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.216355] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8129.253307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8129.531550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.471561] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8144.853800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.031951] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8145.084280] Lustre: 255272:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8145.400695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.033780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8159.081812] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8165.652336] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8172.648962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.878112] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8173.142527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.383807] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8181.946977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8189.489330] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8243.684774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.687033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8243.705078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8245.486205] Lustre: server umount lustre-OST0000 complete [ 8250.155759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8265.291430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.916398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.181662] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8283.259048] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8283.276785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8283.282590] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8283.309985] Lustre: Skipped 1 previous similar message [ 8286.694984] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8329.722962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.722962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8329.722973] LustreError: Skipped 1 previous similar message [ 8329.722992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.734846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.748145] Lustre: Skipped 2 previous similar messages [ 8329.780681] Lustre: Skipped 2 previous similar messages [ 8330.818066] Lustre: server umount lustre-OST0000 complete [ 8337.895798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.898982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8337.921304] Lustre: Skipped 1 previous similar message [ 8341.662277] Lustre: server umount lustre-MDT0000 complete [ 8346.228395] LustreError: 257132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754548 with bad export cookie 8153089202145577771 [ 8346.242146] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.259658] LustreError: 257132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8346.807230] Lustre: server umount lustre-MDT0001 complete [ 8358.500411] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8361.484341] Key type lgssc unregistered [ 8361.894080] LNet: 260703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8361.898807] LNetError: 260703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8361.920329] LNet: Removed LNI 192.168.202.137@tcp [ 8362.837433] Key type .llcrypt unregistered [ 8362.842501] Key type ._llcrypt unregistered [ 8390.442113] Key type ._llcrypt registered [ 8390.448199] Key type .llcrypt registered [ 8390.562149] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8393.272273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.323103] alg: No test for adler32 (adler32-zlib) [ 8394.562378] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8394.994767] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8396.682866] Key type lgssc registered [ 8397.482804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8406.671680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8406.684591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.392907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.757848] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8418.785335] LustreError: 262130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.805851] LustreError: 262130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8423.910907] LustreError: 262131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.159503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.492121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8430.007882] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8440.726616] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8447.476216] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8457.687342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8458.196219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8465.914761] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8474.757948] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8478.715221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8505.315345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8505.332379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.354070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8509.415627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8509.427579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.441777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8510.784784] Lustre: server umount lustre-OST0000 complete [ 8516.583482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8516.587255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.614609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.618727] Lustre: Skipped 1 previous similar message [ 8519.649479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8519.651353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.770085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.775053] Lustre: Skipped 2 previous similar messages [ 8529.379326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8529.548485] Lustre: server umount lustre-MDT0000 complete [ 8529.890848] LustreError: 262130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.902745] LustreError: 262130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8533.927177] LustreError: 262866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754736 with bad export cookie 16121068812864273188 [ 8533.944438] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.276193] Lustre: server umount lustre-MDT0001 complete [ 8544.930484] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8548.206678] Key type lgssc unregistered [ 8548.584794] LNet: 266442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8548.628591] LNetError: 266442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8548.649161] LNet: Removed LNI 192.168.202.137@tcp [ 8549.537378] Key type .llcrypt unregistered [ 8549.539873] Key type ._llcrypt unregistered [ 8568.640135] Key type ._llcrypt registered [ 8568.644336] Key type .llcrypt registered [ 8568.855944] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8569.843079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8569.854412] alg: No test for adler32 (adler32-zlib) [ 8570.850377] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8571.013409] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8572.672571] Key type lgssc registered [ 8573.880472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8582.946754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8582.984917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.707990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.979989] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8593.901849] LustreError: 267664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8593.948039] LustreError: 267664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8599.011166] LustreError: 267663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.855348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.274559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8606.559356] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8615.902829] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8622.190516] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8630.188604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8630.492296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8637.330087] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8647.203783] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8654.084472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8673.764044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8673.770745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8673.796488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.808220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8674.817340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.818781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.694727] Lustre: server umount lustre-OST0000 complete [ 8684.002159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8684.010216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.021067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.025274] Lustre: Skipped 1 previous similar message [ 8688.340423] Lustre: server umount lustre-MDT0000 complete [ 8690.153652] LustreError: 267658:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8690.169311] LustreError: 267658:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8692.180661] LustreError: 267645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754894 with bad export cookie 5428597316031926281 [ 8692.186654] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.191529] LustreError: 267645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8692.441147] Lustre: server umount lustre-MDT0001 complete [ 8704.932745] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8707.914917] Key type lgssc unregistered [ 8708.162843] LNet: 271369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8708.169801] LNetError: 271369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8708.183201] LNet: Removed LNI 192.168.202.137@tcp [ 8709.562932] Key type .llcrypt unregistered [ 8709.565410] Key type ._llcrypt unregistered [ 8741.337878] Key type ._llcrypt registered [ 8741.340336] Key type .llcrypt registered [ 8741.555160] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8743.182784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8743.219214] alg: No test for adler32 (adler32-zlib) [ 8744.662339] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8744.881297] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8746.568320] Key type lgssc registered [ 8748.017352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8757.961299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8757.984515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.534417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.436840] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8769.007384] LustreError: 272786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.038836] LustreError: 272786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8774.115687] LustreError: 272787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.467808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.063331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8781.855384] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8789.918468] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8797.833601] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.071453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8806.548760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.960496] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8824.399828] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8829.113084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8858.088201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8858.097679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8858.120978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.131306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8860.147113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.173100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.183475] Lustre: Skipped 1 previous similar message [ 8862.769763] Lustre: server umount lustre-OST0000 complete [ 8869.345851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8869.361562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.386678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8875.491756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8875.494766] Lustre: Skipped 3 previous similar messages [ 8880.611465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8880.616734] Lustre: Skipped 1 previous similar message [ 8881.120881] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8881.489283] Lustre: server umount lustre-MDT0000 complete [ 8885.730179] LustreError: 272787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.745728] LustreError: 272787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8886.606887] LustreError: 272767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755088 with bad export cookie 9659851882996516958 [ 8886.621594] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.626470] LustreError: 272767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8887.265807] Lustre: server umount lustre-MDT0001 complete [ 8897.544638] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 8900.831441] Key type lgssc unregistered [ 8901.143316] LNet: 277100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8901.158550] LNetError: 277100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8902.182804] LNet: Removed LNI 192.168.202.137@tcp [ 8903.555626] Key type .llcrypt unregistered [ 8903.559883] Key type ._llcrypt unregistered [ 8921.433797] Key type ._llcrypt registered [ 8921.435481] Key type .llcrypt registered [ 8921.545495] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 8922.877382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8922.970230] alg: No test for adler32 (adler32-zlib) [ 8924.064864] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 8924.310194] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 8925.977387] Key type lgssc registered [ 8927.247682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8936.158738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8936.176964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.919820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8943.058471] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8943.077766] LustreError: 278338:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.117342] LustreError: 278338:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8948.194058] LustreError: 278340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8953.314446] LustreError: 278342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8953.721870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8954.096559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8959.986775] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8968.472946] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8974.216235] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8981.747611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.172854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8983.239991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8989.778513] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 8999.400700] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9014.243020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9014.268110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.295440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9014.536307] Lustre: server umount lustre-OST0000 complete [ 9020.898265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9020.908599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9020.920300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9025.378471] Lustre: server umount lustre-MDT0000 complete [ 9026.532718] LustreError: 278341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9026.557348] LustreError: 278341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9030.000202] LustreError: 278305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755232 with bad export cookie 17746928417591978465 [ 9030.003275] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9030.011564] LustreError: 278305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9040.566955] Lustre: server umount lustre-MDT0001 complete [ 9054.207339] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9058.515719] Key type lgssc unregistered [ 9058.851671] LNet: 282045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9058.863102] LNetError: 282045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9059.880262] LNet: Removed LNI 192.168.202.137@tcp [ 9061.613638] Key type .llcrypt unregistered [ 9061.616420] Key type ._llcrypt unregistered [ 9078.607646] Key type ._llcrypt registered [ 9078.610545] Key type .llcrypt registered [ 9078.790780] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9099.786529] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9119.436678] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9121.287058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9121.337601] alg: No test for adler32 (adler32-zlib) [ 9122.402238] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 9122.677766] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9124.368253] Key type lgssc registered [ 9125.844913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9171.392855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9171.415121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.837355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9172.910703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9173.135290] Lustre: lustre-MDT0000: new disk, initializing [ 9173.277076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.306241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9178.738771] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9193.581899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.795345] Lustre: 285603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9193.906485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9193.910064] Lustre: Skipped 1 previous similar message [ 9193.997611] Lustre: lustre-MDT0001: new disk, initializing [ 9194.115647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9194.160859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9194.175432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9199.796534] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9209.934310] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9215.455575] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9222.271114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9222.613366] Lustre: lustre-OST0000: new disk, initializing [ 9222.620575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9222.627693] Lustre: 287570:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9222.822214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.458167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9229.478883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9229.655307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9229.726397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9240.098364] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9245.514469] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9245.719454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9250.759326] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9250.990629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9255.403505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.413252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9255.413703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.443095] Lustre: Skipped 1 previous similar message [ 9255.476742] Lustre: Skipped 1 previous similar message [ 9260.516835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9260.527425] Lustre: Skipped 1 previous similar message [ 9265.637543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9267.168379] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9267.354446] Lustre: server umount lustre-OST0000 complete [ 9275.876113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9275.879436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9275.893488] LustreError: Skipped 1 previous similar message [ 9275.894904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9275.911654] Lustre: Skipped 2 previous similar messages [ 9277.797694] Lustre: server umount lustre-MDT0000 complete [ 9281.994852] LustreError: 285594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755484 with bad export cookie 17100618987559848178 [ 9282.036472] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9282.561532] Lustre: server umount lustre-MDT0001 complete [ 9291.557182] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:58:13 (1786755493) [ 9298.646069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.128037] LustreError: 289982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.145906] LustreError: 289982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9299.226491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9304.546289] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9304.632771] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9309.674317] LustreError: 289982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9314.793605] LustreError: 289983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9315.065409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9315.393101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9320.402874] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9330.268733] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9338.020857] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9346.360044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9346.840330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9355.408987] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9363.199570] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9379.816985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9379.818893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9379.835276] Lustre: Skipped 2 previous similar messages [ 9379.840357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9379.848886] Lustre: Skipped 1 previous similar message [ 9385.737044] Lustre: server umount lustre-OST0000 complete [ 9390.067285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.070593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9390.074597] Lustre: Skipped 1 previous similar message [ 9390.094269] Lustre: Skipped 3 previous similar messages [ 9392.046579] Lustre: server umount lustre-MDT0000 complete [ 9395.169683] LustreError: 290710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.198397] LustreError: 290710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9395.655559] LustreError: 289964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755597 with bad export cookie 17100618987559849060 [ 9395.665142] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9395.666594] LustreError: 289964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9395.967735] Lustre: server umount lustre-MDT0001 complete [ 9406.822600] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9409.569436] Key type lgssc unregistered [ 9409.856464] LNet: 293543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9409.863878] LNetError: 293543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9409.884293] LNet: Removed LNI 192.168.202.137@tcp [ 9410.616777] Key type .llcrypt unregistered [ 9410.619223] Key type ._llcrypt unregistered [ 9433.827845] Key type ._llcrypt registered [ 9433.831869] Key type .llcrypt registered [ 9433.928600] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9448.392449] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9449.453895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9449.573117] alg: No test for adler32 (adler32-zlib) [ 9450.682480] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 9450.856388] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9452.507976] Key type lgssc registered [ 9453.760073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9505.470188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9505.536110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9506.845391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9506.882706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9507.010883] Lustre: lustre-MDT0000: new disk, initializing [ 9507.078112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.099858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9512.212736] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9523.392528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.574053] Lustre: 297382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9523.619912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9523.624347] Lustre: Skipped 1 previous similar message [ 9523.744487] Lustre: lustre-MDT0001: new disk, initializing [ 9523.892964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9523.952096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9523.959541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9528.195969] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9537.975156] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9542.954196] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9549.660963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9550.084705] Lustre: lustre-OST2710: new disk, initializing [ 9550.089291] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9550.100321] Lustre: 299351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9550.209097] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9557.400151] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9557.634116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9557.664773] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9557.854362] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9566.637877] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9572.865071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9573.036608] Lustre: lustre-OST03e8: new disk, initializing [ 9573.050272] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9573.058714] Lustre: 300414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9573.182068] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9579.100557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9579.119079] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9579.214108] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9579.548272] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9587.624105] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9593.936807] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9594.228071] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9598.507397] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9598.714622] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9602.725688] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9602.895452] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9607.664995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9607.887190] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9611.242760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9611.253181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.278390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.335695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9615.336986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.358260] Lustre: Skipped 1 previous similar message [ 9615.382985] Lustre: Skipped 1 previous similar message [ 9617.703576] Lustre: server umount lustre-MDT0000 complete [ 9621.470888] LustreError: 297373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755823 with bad export cookie 4247126944968288142 [ 9621.473997] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9621.482740] LustreError: 297373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9621.741639] Lustre: server umount lustre-MDT0001 complete [ 9635.943153] Lustre: server umount lustre-OST2710 complete [ 9649.398655] Lustre: server umount lustre-OST03e8 complete [ 9666.648793] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [ 9676.342267] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9709.810327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9710.039973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9710.060318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9710.116351] Lustre: lustre-MDT0000: new disk, initializing [ 9710.161403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.173340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9714.830545] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9725.420693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.579523] Lustre: 305662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9725.657288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9725.664820] Lustre: Skipped 1 previous similar message [ 9725.839449] Lustre: lustre-MDT0001: new disk, initializing [ 9726.006341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9726.099239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9726.108218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9732.147440] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9743.800665] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9749.457710] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9757.795566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9758.057402] Lustre: lustre-OST0000: new disk, initializing [ 9758.060892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9758.066507] Lustre: 307634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9758.173460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.447936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9764.478595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9764.588053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9765.246752] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9775.582215] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9781.263991] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9781.521458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9786.604732] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9786.932362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9790.436562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9790.437156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9790.460688] LustreError: Skipped 1 previous similar message [ 9790.461445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9790.461449] Lustre: Skipped 2 previous similar messages [ 9790.487576] Lustre: Skipped 2 previous similar messages [ 9795.565943] Lustre: server umount lustre-OST0000 complete [ 9800.687424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9800.699411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9800.728047] Lustre: Skipped 1 previous similar message [ 9800.747329] Lustre: Skipped 2 previous similar messages [ 9805.793571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9805.801615] Lustre: Skipped 1 previous similar message [ 9806.026721] Lustre: server umount lustre-MDT0000 complete [ 9809.965448] LustreError: 306588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.992880] LustreError: 306588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9810.332113] LustreError: 307632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756012 with bad export cookie 4247126944968289822 [ 9810.343698] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9810.350748] LustreError: 307632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9810.934399] Lustre: server umount lustre-MDT0001 complete [ 9820.273342] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:07:01 (1786756021) [ 9821.657302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9823.656666] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:07:05 (1786756025) [ 9827.859480] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9855.213536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.638420] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9855.655352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9855.887318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.177146] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9871.226390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.430444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9871.454667] Lustre: 311492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9877.481879] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [ 9899.196512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9899.421610] Lustre: Denying initial registration attempt from nid 192.168.202.137@tcp, specified as failover [ 9899.431423] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9899.437156] LustreError: 312948:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9899.441063] LustreError: 312948:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9899.444142] LustreError: 312948:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9899.511479] Lustre: server umount lustre-OST0000 complete [ 9899.513865] LustreError: 312948:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9907.682656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9907.690106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.716396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9913.539352] Lustre: server umount lustre-MDT0000 complete [ 9914.344101] LustreError: 311499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.355051] LustreError: 311499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9917.308623] LustreError: 311484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756119 with bad export cookie 4247126944968290704 [ 9917.317279] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.332783] LustreError: 311484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9917.671690] Lustre: server umount lustre-MDT0001 complete [ 9928.320275] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [ 9931.559689] Key type lgssc unregistered [ 9931.839181] LNet: 314070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9931.857656] LNetError: 314070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9932.903249] LNet: Removed LNI 192.168.202.137@tcp [ 9933.790580] Key type .llcrypt unregistered [ 9933.795310] Key type ._llcrypt unregistered [ 9948.300504] Key type ._llcrypt registered [ 9948.310024] Key type .llcrypt registered [ 9948.423525] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [ 9949.178135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9949.199610] alg: No test for adler32 (adler32-zlib) [ 9950.240801] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [ 9950.570724] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [ 9952.232539] Key type lgssc registered [ 9953.645775] Lustre: Echo OBD driver; http://www.lustre.org/ [10002.342640] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10012.650579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10012.679506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.902726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10013.918452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10014.043491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10018.491824] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10027.068554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.122686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10027.144883] Lustre: 316831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10027.332775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10032.154982] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10040.441127] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10047.906228] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10069.294064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.706453] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10070.125871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.487170] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10087.065760] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10092.538347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.547647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10092.548690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.566809] Lustre: Skipped 1 previous similar message [10096.049993] Lustre: server umount lustre-OST0000 complete [10102.775510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10102.776983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10102.803071] Lustre: Skipped 1 previous similar message [10102.822742] Lustre: Skipped 2 previous similar messages [10106.497068] Lustre: server umount lustre-MDT0000 complete [10110.569791] LustreError: 316825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756312 with bad export cookie 12964073382208719593 [10110.589863] LustreError: 316825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10110.592916] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10110.928605] Lustre: server umount lustre-MDT0001 complete [10123.167559] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10125.831976] Key type lgssc unregistered [10126.106778] LNet: 320413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10126.114352] LNetError: 320413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10126.125521] LNet: Removed LNI 192.168.202.137@tcp [10127.238112] Key type .llcrypt unregistered [10127.241610] Key type ._llcrypt unregistered [10158.261504] Key type ._llcrypt registered [10158.263835] Key type .llcrypt registered [10158.474821] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10159.941919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10160.192938] alg: No test for adler32 (adler32-zlib) [10161.349258] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [10161.591636] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10163.304187] Key type lgssc registered [10164.631355] Lustre: Echo OBD driver; http://www.lustre.org/ [10174.333468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10174.347358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.028279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10181.478683] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10186.217774] LustreError: 321833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10186.240742] LustreError: 321833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10191.336187] LustreError: 321834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.367922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.786140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10195.982786] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10204.776088] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10211.520216] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10220.604501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10221.073715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10227.718916] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10235.599698] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10246.637483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10246.642372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10246.646021] Lustre: Skipped 1 previous similar message [10246.667670] Lustre: Skipped 2 previous similar messages [10248.838986] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [10251.748363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10251.764029] Lustre: Skipped 2 previous similar messages [10252.530580] Lustre: server umount lustre-MDT0000 complete [10253.906307] LustreError: 321828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.866038] LustreError: 323323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10256.904573] LustreError: 323323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10257.801206] LustreError: 321815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756459 with bad export cookie 1730336768403997754 [10257.824368] LustreError: 321815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10257.826480] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10259.054888] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [10261.986346] LustreError: 321828:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.987798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10262.023591] LustreError: 321828:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10262.055852] Lustre: Skipped 1 previous similar message [10264.150383] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [10264.152931] Lustre: Skipped 2 previous similar messages [10270.341701] LustreError: 321830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10270.386058] LustreError: 321830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10272.225429] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10272.229848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10272.245602] Lustre: Skipped 3 previous similar messages [10272.562288] Lustre: server umount lustre-MDT0001 complete [10280.595151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10291.017837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10302.292918] LustreError: 323666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10302.322633] LustreError: 323666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10302.402072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10302.472551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10307.635952] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10316.809788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.174598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10322.075187] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10327.546343] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10330.741619] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10332.647804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10338.164633] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10339.762660] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10348.001439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10348.007137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.026866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10348.031225] Lustre: Skipped 1 previous similar message [10351.652503] Lustre: server umount lustre-OST0000 complete [10356.196734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.214460] Lustre: Skipped 1 previous similar message [10361.982943] Lustre: server umount lustre-MDT0000 complete [10365.646166] LustreError: 321815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756567 with bad export cookie 1730336768404004901 [10365.659265] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.968645] Lustre: server umount lustre-MDT0001 complete [10378.009637] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10381.350337] Key type lgssc unregistered [10381.616134] LNet: 328106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10381.623591] LNetError: 328106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10381.635760] LNet: Removed LNI 192.168.202.137@tcp [10382.416768] Key type .llcrypt unregistered [10382.418911] Key type ._llcrypt unregistered [10409.875841] Key type ._llcrypt registered [10409.881840] Key type .llcrypt registered [10410.028787] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10411.345030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10411.443799] alg: No test for adler32 (adler32-zlib) [10412.451798] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [10412.721707] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10414.448260] Key type lgssc registered [10415.722530] Lustre: Echo OBD driver; http://www.lustre.org/ [10423.503156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10423.532775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.996042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10429.660923] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10435.563939] LustreError: 329524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10435.593582] LustreError: 329524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10438.550932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10438.888946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10444.143743] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10452.792645] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10460.200446] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10464.738126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10464.747909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10464.766699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10468.841066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10468.842541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10468.868227] Lustre: Skipped 1 previous similar message [10470.083490] Lustre: server umount lustre-MDT0000 complete [10473.303869] LustreError: 329505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756675 with bad export cookie 13091444234772389027 [10473.305323] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10473.316620] LustreError: 329505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10473.780906] Lustre: server umount lustre-MDT0001 complete [10488.972492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.309871] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10489.360309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10489.623777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10494.740194] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10502.641989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10502.754217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10502.780770] Lustre: 332115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10502.998428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10507.305730] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10514.816847] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10521.209752] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.749270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10531.193284] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10531.207975] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10531.229112] LustreError: 333997:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10531.238523] LustreError: 333997:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10531.252133] LustreError: 333997:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10531.340852] Lustre: server umount lustre-OST0000 complete [10531.342848] LustreError: 333997:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10541.748497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10542.037898] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10542.262624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10548.869385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10549.186691] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10557.921596] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10568.338720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.444483] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10568.533950] Lustre: lustre-OST0001: new disk, initializing [10568.537135] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10568.542728] Lustre: 335539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10568.665758] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10577.720303] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10579.161980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10579.183665] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10579.338414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10586.938303] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10594.797105] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.817247] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10594.821106] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10594.827875] LustreError: Skipped 1 previous similar message [10594.843535] Lustre: Skipped 1 previous similar message [10599.914672] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10599.924461] Lustre: Skipped 1 previous similar message [10605.027496] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10605.036841] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10605.053027] Lustre: Skipped 1 previous similar message [10605.236193] Lustre: server umount lustre-OST0001 complete [10610.158615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.160802] LustreError: 334465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.166940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10610.170052] Lustre: Skipped 2 previous similar messages [10610.204215] LustreError: 334465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10615.269432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10615.273024] LustreError: 334463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10615.279990] Lustre: Skipped 3 previous similar messages [10615.302293] LustreError: 334463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10615.621703] Lustre: server umount lustre-OST0000 complete [10620.389066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10620.421201] Lustre: Skipped 1 previous similar message [10625.869836] Lustre: server umount lustre-MDT0000 complete [10630.636513] LustreError: 333309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.672643] LustreError: 333309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10630.955628] LustreError: 332105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756833 with bad export cookie 13091444234772389580 [10630.989723] LustreError: 332105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10630.992330] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10631.968977] Lustre: server umount lustre-MDT0001 complete [10648.988273] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [10652.450751] Key type lgssc unregistered [10652.737676] LNet: 337527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10652.744401] LNetError: 337527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10652.762459] LNet: Removed LNI 192.168.202.137@tcp [10653.663948] Key type .llcrypt unregistered [10653.667530] Key type ._llcrypt unregistered [10740.856783] Key type ._llcrypt registered [10740.861756] Key type .llcrypt registered [10740.983945] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [10755.196942] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [10756.297270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10756.401372] alg: No test for adler32 (adler32-zlib) [10757.526975] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [10757.812748] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [10759.441736] Key type lgssc registered [10760.697784] Lustre: Echo OBD driver; http://www.lustre.org/ [10794.722324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10794.774174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.024306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10796.056976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10796.125594] Lustre: lustre-MDT0000: new disk, initializing [10796.230085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10796.242552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10801.023971] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10812.168388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.377552] Lustre: 342462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10812.468357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10812.476421] Lustre: Skipped 1 previous similar message [10812.598273] Lustre: lustre-MDT0001: new disk, initializing [10812.681996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10812.713098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10812.746877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10817.863420] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10827.499823] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10833.917601] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10842.114842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.606348] Lustre: lustre-OST0000: new disk, initializing [10842.609530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10842.614242] Lustre: 344433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10842.747684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10850.538446] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10851.950899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10851.964641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10852.110374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10861.308800] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10867.273707] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10867.509097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10871.557986] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10871.854228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10877.926883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.931376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10877.944343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10877.945346] Lustre: Skipped 1 previous similar message [10877.972674] LustreError: Skipped 1 previous similar message [10879.822220] Lustre: server umount lustre-OST0000 complete [10884.583165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10884.596758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10884.640041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10884.658459] Lustre: Skipped 1 previous similar message [10888.162623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10888.164125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10890.601404] Lustre: server umount lustre-MDT0000 complete [10894.832601] LustreError: 342456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757096 with bad export cookie 5010788184323925036 [10894.842500] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10894.850495] LustreError: 342456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10895.497362] Lustre: server umount lustre-MDT0001 complete [10905.887410] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:25:07 (1786757107) [10916.227843] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:25:17 (1786757117) [10929.742903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.571962] LustreError: 347188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.605327] LustreError: 347188:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10930.659035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10935.110560] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10935.777309] LustreError: 347189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.897930] LustreError: 347188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.753678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10944.213190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10950.032329] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10958.774936] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.602468] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.796690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10977.040477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10983.913711] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [10993.942326] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11014.454482] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11016.728079] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11018.678969] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11020.766538] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11023.186837] Lustre: Failing over lustre-MDT0000 [11023.439718] Lustre: server umount lustre-MDT0000 complete [11025.890800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11025.904670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.926287] LustreError: 347188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.149867] LustreError: 349232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.166809] LustreError: 349232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11042.393978] LustreError: 349233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.412056] LustreError: 349233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11043.329299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.445971] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11043.810615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11043.913421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11048.529957] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11048.535571] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11048.950940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11048.979528] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11049.030910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11058.007248] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11059.529845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11063.148929] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11065.349864] Lustre: Failing over lustre-MDT0000 [11065.587258] Lustre: server umount lustre-MDT0000 complete [11069.043748] LustreError: 349232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11069.076128] LustreError: 349232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11069.424530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11069.440045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11069.455535] Lustre: Skipped 4 previous similar messages [11084.346488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11084.529254] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11084.859570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11084.927792] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11085.433359] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11089.951951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11089.957557] Lustre: Skipped 2 previous similar messages [11090.041037] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11090.191805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11090.601949] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11105.793550] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11107.656539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11115.493051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11115.501641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11115.512957] Lustre: Skipped 2 previous similar messages [11115.523963] Lustre: Skipped 3 previous similar messages [11120.609898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11120.987453] Lustre: server umount lustre-MDT0000 complete [11125.002705] LustreError: 347170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757327 with bad export cookie 5010788184323927752 [11125.008599] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11125.017818] LustreError: 347170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11125.758548] Lustre: server umount lustre-MDT0001 complete [11141.661545] Lustre: server umount lustre-OST0000 complete [11159.239931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.774963] LustreError: 353556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.807716] LustreError: 353556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [11159.892366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.877588] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11175.250131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11181.056179] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11190.526293] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11197.859973] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11207.199425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.428869] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11226.868210] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11229.487327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11240.946682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11240.953978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11240.966693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11240.972422] Lustre: Skipped 2 previous similar messages [11246.817536] Lustre: server umount lustre-OST0000 complete [11252.194840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11252.206608] LustreError: Skipped 1 previous similar message [11257.075371] Lustre: server umount lustre-MDT0000 complete [11260.388517] LustreError: 354284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.428271] LustreError: 354284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11261.216533] LustreError: 353536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757463 with bad export cookie 5010788184323928501 [11261.219884] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11261.227607] LustreError: 353536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11261.557962] Lustre: server umount lustre-MDT0001 complete [11273.457222] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11275.977760] Key type lgssc unregistered [11276.191613] LNet: 357114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11276.198283] LNetError: 357114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11276.218333] LNet: Removed LNI 192.168.202.137@tcp [11277.103903] Key type .llcrypt unregistered [11277.105866] Key type ._llcrypt unregistered [11313.373773] Key type ._llcrypt registered [11313.376338] Key type .llcrypt registered [11313.587731] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11314.991277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11315.195301] alg: No test for adler32 (adler32-zlib) [11316.304995] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11316.517995] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11318.274105] Key type lgssc registered [11319.359196] Lustre: Echo OBD driver; http://www.lustre.org/ [11327.509459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11327.530303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11329.073577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11334.669724] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11339.234342] LustreError: 358633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.263898] LustreError: 358633:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11344.353457] LustreError: 358634:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.421655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.741943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11349.605646] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11359.104474] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11368.703722] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11377.679487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11378.137518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11385.304690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11394.423296] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11399.696439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11412.265211] Lustre: server umount lustre-MDT0000 complete [11415.021295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.030617] LustreError: 358629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11415.038493] Lustre: Skipped 2 previous similar messages [11415.068141] LustreError: 358629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11416.362904] LustreError: 358613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757618 with bad export cookie 16707989340290143530 [11416.365031] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11416.386602] LustreError: 358613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11416.696293] Lustre: server umount lustre-MDT0001 complete [11431.292552] Lustre: server umount lustre-OST0000 complete [11446.620961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.106072] LustreError: 362372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.178700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11452.386390] LustreError: 362373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11452.439494] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11462.212820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.629379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11462.841337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11469.679690] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11479.579622] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11487.181630] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11495.180700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11495.771812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11502.352637] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11512.189071] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11517.312755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11532.780962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11532.795901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11532.819159] Lustre: Skipped 2 previous similar messages [11532.847774] Lustre: Skipped 2 previous similar messages [11536.269623] Lustre: server umount lustre-MDT0000 complete [11537.893538] LustreError: 362373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11537.934960] LustreError: 362373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11541.759597] LustreError: 365055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757743 with bad export cookie 16707989340290144643 [11541.774363] LustreError: 365055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11541.778305] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11542.391515] Lustre: server umount lustre-MDT0001 complete [11558.847813] Lustre: server umount lustre-OST0000 complete [11576.739604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.366784] LustreError: 366285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.453114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.788418] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11593.878608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.400815] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11610.571191] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11619.193957] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11628.301027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11628.583378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11628.591782] Lustre: Skipped 1 previous similar message [11637.154972] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11647.095187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11649.273915] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11657.704185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11657.706594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11657.715668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11657.734329] Lustre: Skipped 1 previous similar message [11662.824082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11662.830608] Lustre: Skipped 1 previous similar message [11663.105576] Lustre: server umount lustre-OST0000 complete [11667.941207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11667.945320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.952385] Lustre: Skipped 1 previous similar message [11667.972880] Lustre: Skipped 2 previous similar messages [11673.627579] Lustre: server umount lustre-MDT0000 complete [11678.181391] LustreError: 368584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.192694] LustreError: 368584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11678.430836] LustreError: 368131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757880 with bad export cookie 16707989340290145665 [11678.431612] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11678.443971] LustreError: 368131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11678.991507] Lustre: server umount lustre-MDT0001 complete [11691.924418] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11695.435713] Key type lgssc unregistered [11695.876794] LNet: 369826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11695.885899] LNetError: 369826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11695.905342] LNet: Removed LNI 192.168.202.137@tcp [11696.824702] Key type .llcrypt unregistered [11696.827276] Key type ._llcrypt unregistered [11724.323697] Key type ._llcrypt registered [11724.326272] Key type .llcrypt registered [11724.567543] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11726.709723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11726.760531] alg: No test for adler32 (adler32-zlib) [11728.072923] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11728.469204] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11730.161190] Key type lgssc registered [11731.952988] Lustre: Echo OBD driver; http://www.lustre.org/ [11741.580919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11741.602208] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.609869] LustreError: 371271:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11741.629226] LustreError: 371271:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11741.647449] LustreError: 371271:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11741.663493] LustreError: 371271:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11741.688302] LustreError: 371271:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11750.120724] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11750.130718] LustreError: 371669:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11750.179390] LustreError: 371669:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11750.195678] LustreError: 371669:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11750.202334] LustreError: 371669:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11750.210210] LustreError: 371669:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11776.034699] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [11779.280415] Key type lgssc unregistered [11779.765033] LNet: 372676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11779.779348] LNetError: 372676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11779.801933] LNet: Removed LNI 192.168.202.137@tcp [11780.811240] Key type .llcrypt unregistered [11780.815914] Key type ._llcrypt unregistered [11800.234264] Key type ._llcrypt registered [11800.235893] Key type .llcrypt registered [11800.400450] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [11816.776596] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [11818.164210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11818.212386] alg: No test for adler32 (adler32-zlib) [11819.420433] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [11819.766892] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [11821.504210] Key type lgssc registered [11822.727297] Lustre: Echo OBD driver; http://www.lustre.org/ [11859.400868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11859.428930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11860.731277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11860.797647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11860.910153] Lustre: lustre-MDT0000: new disk, initializing [11861.050657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11861.071415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11866.297979] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11877.949451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11878.112418] Lustre: 375834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11878.190881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11878.194503] Lustre: Skipped 1 previous similar message [11878.262891] Lustre: lustre-MDT0001: new disk, initializing [11878.351440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11878.391276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11878.402444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11883.649859] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11894.820858] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11901.006735] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11908.669226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.135577] Lustre: lustre-OST0000: new disk, initializing [11909.140216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11909.147132] Lustre: 377804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11909.244206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11915.915610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11915.933970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11916.212134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11918.160658] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11929.296206] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11935.152994] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11935.476244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11941.042969] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11941.353256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11946.977619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11946.992483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.007470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11949.905275] Lustre: server umount lustre-OST0000 complete [11955.172135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11955.181035] LustreError: Skipped 1 previous similar message [11955.182917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11955.200229] Lustre: Skipped 1 previous similar message [11955.216755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11955.221137] Lustre: Skipped 1 previous similar message [11957.218276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.220715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11957.250079] Lustre: Skipped 1 previous similar message [11960.180908] Lustre: server umount lustre-MDT0000 complete [11964.860507] LustreError: 375827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758166 with bad export cookie 18269716167348947305 [11964.869841] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11965.136878] Lustre: server umount lustre-MDT0001 complete [11975.190369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11977.445908] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:42:58 (1786758178) [11984.579476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.241144] LustreError: 380265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.278331] LustreError: 380265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11985.353882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11990.498556] LustreError: 380266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11990.768280] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [11995.617995] LustreError: 380265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12000.488521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12000.742756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12000.865392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12005.554701] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12014.496144] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12021.036737] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12029.881897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.572385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12038.158432] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12048.407508] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12057.282506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12057.548963] Lustre: lustre-OST0001: new disk, initializing [12057.552906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12057.560925] Lustre: 383104:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12057.652821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12063.355201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12063.373796] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12063.490682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12064.538759] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12075.272918] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12079.100908] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12079.109755] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.156210] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12079.170769] Lustre: Skipped 1 previous similar message [12085.170846] Lustre: server umount lustre-OST0001 complete [12087.376278] LustreError: 382090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.438989] LustreError: 383113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.450819] LustreError: 383113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12104.683426] LustreError: 382091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12104.696601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.698535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12104.698549] Lustre: Skipped 3 previous similar messages [12104.701844] LustreError: 382091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12104.758761] Lustre: Skipped 2 previous similar messages [12108.597665] Lustre: server umount lustre-OST0000 complete [12113.378115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12113.382168] LustreError: Skipped 1 previous similar message [12113.386406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.314383] Lustre: server umount lustre-MDT0000 complete [12124.157055] LustreError: 380261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.181720] LustreError: 380261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12124.630295] LustreError: 380247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758326 with bad export cookie 18269716167348948187 [12124.647513] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12124.649646] LustreError: 380247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12131.084877] Lustre: server umount lustre-MDT0001 complete [12144.579941] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12147.795828] Key type lgssc unregistered [12148.067706] LNet: 385094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12148.074789] LNetError: 385094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12148.118885] LNet: Removed LNI 192.168.202.137@tcp [12148.976509] Key type .llcrypt unregistered [12148.978648] Key type ._llcrypt unregistered [12206.632383] Key type ._llcrypt registered [12206.633924] Key type .llcrypt registered [12206.769508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12208.282898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12208.333375] alg: No test for adler32 (adler32-zlib) [12209.425895] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [12209.783985] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12211.578630] Key type lgssc registered [12212.869871] Lustre: Echo OBD driver; http://www.lustre.org/ [12221.474040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12221.493701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12222.879054] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12222.923072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12223.200186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12228.337896] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12238.114130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.274254] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12238.324853] Lustre: 387899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12238.694983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12244.546437] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12253.154486] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12261.189748] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12265.446141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12265.475726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12265.488974] Lustre: Skipped 1 previous similar message [12270.571249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.599156] Lustre: Skipped 1 previous similar message [12271.272317] Lustre: server umount lustre-MDT0000 complete [12276.993763] LustreError: 387890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758479 with bad export cookie 4198074279998842539 [12277.022133] LustreError: 387890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12277.027537] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12278.092725] Lustre: server umount lustre-MDT0001 complete [12289.363125] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:48:10 (1786758490) [12297.586450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.292785] LustreError: 390411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.312055] LustreError: 390411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12298.365876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12303.843749] LustreError: 390412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12304.315964] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12308.966685] LustreError: 390411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12314.081893] LustreError: 390412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12315.245489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12315.900492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12322.773233] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12334.129283] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12343.109092] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12351.737731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12351.954869] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12352.199476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12359.916335] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12369.142768] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12378.116833] Lustre: Permanently deactivating lustre-OST0000 [12378.137716] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12384.371036] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12386.314898] LustreError: 393353:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12386.321899] LustreError: 393353:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12386.325603] LustreError: 393353:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12390.060679] LustreError: 393402:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12390.069377] LustreError: 393402:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12398.304904] Lustre: server umount lustre-OST0000 complete [12400.836480] LustreError: 393652:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12400.838772] LustreError: 393652:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12403.178313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12403.192477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12403.216832] Lustre: Skipped 1 previous similar message [12403.226111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12407.781290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12407.792502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.085109] Lustre: server umount lustre-MDT0000 complete [12412.900866] LustreError: 390412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12412.912762] LustreError: 390412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12416.214989] LustreError: 390392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758618 with bad export cookie 4198074279998843120 [12416.224809] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12416.229357] LustreError: 390392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12426.879857] Lustre: server umount lustre-MDT0001 complete [12437.356552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.567866] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12444.314808] LustreError: 394812:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12452.256832] Lustre: 395152:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.137@tcp [12455.489869] Lustre: server umount MGS complete [12468.489195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12468.905802] LustreError: 395877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.933647] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12468.940600] Lustre: Skipped 1 previous similar message [12468.988683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12473.740874] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12484.140857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.491907] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12491.318543] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12501.681159] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12509.519374] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12512.042279] Lustre: Permanently reactivating lustre-OST0000 [12512.056927] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12512.065384] Lustre: Skipped 2 previous similar messages [12539.872397] LustreError: 397626:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12548.742426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12549.276487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12549.290661] Lustre: Skipped 1 previous similar message [12557.399234] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12567.913818] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12574.224411] Lustre: server umount lustre-OST0000 complete [12581.375254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12581.390823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12581.412208] Lustre: Skipped 1 previous similar message [12581.439717] Lustre: Skipped 2 previous similar messages [12588.717429] Lustre: server umount lustre-MDT0000 complete [12591.586710] Lustre: 387220:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758777/real 1786758777] req@000000004ca5a902 x1873551987676032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786758793 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12591.590883] LustreError: 395878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12591.636923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12591.678016] LustreError: 395878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12592.612257] Lustre: 387222:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758778/real 1786758778] req@000000007750abca x1873551987676800/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786758794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12593.347280] LustreError: 395857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758795 with bad export cookie 4198074279998844170 [12593.364950] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12594.088399] Lustre: server umount lustre-MDT0001 complete [12607.184487] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12610.988492] Key type lgssc unregistered [12611.465704] LNet: 400197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12611.472677] LNetError: 400197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12611.501609] LNet: Removed LNI 192.168.202.137@tcp [12612.672498] Key type .llcrypt unregistered [12612.674682] Key type ._llcrypt unregistered [12637.111571] Key type ._llcrypt registered [12637.114427] Key type .llcrypt registered [12637.301911] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [12655.715456] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [12656.512469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12656.551783] alg: No test for adler32 (adler32-zlib) [12657.648934] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [12658.179412] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [12659.920393] Key type lgssc registered [12661.738981] Lustre: Echo OBD driver; http://www.lustre.org/ [12711.697723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12711.738173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12713.084796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12713.128233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12713.214804] Lustre: lustre-MDT0000: new disk, initializing [12713.320527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12713.341871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12718.138472] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12730.183942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12730.352225] Lustre: 403360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12730.404330] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12730.407940] Lustre: Skipped 1 previous similar message [12730.475365] Lustre: lustre-MDT0001: new disk, initializing [12730.603749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12730.669126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12730.684954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12735.210992] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12744.780510] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12750.702575] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12758.762422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12759.059736] Lustre: lustre-OST0000: new disk, initializing [12759.064369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12759.071952] Lustre: 405326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12759.187566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12764.780235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12764.788728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12764.911791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12765.523295] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12776.795872] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12781.673669] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12781.909210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12787.079756] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12787.328791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12790.777218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12790.788775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12790.818966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12790.832397] LustreError: Skipped 1 previous similar message [12795.860935] Lustre: server umount lustre-OST0000 complete [12800.038613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12800.047288] Lustre: Skipped 1 previous similar message [12800.056558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12800.064813] Lustre: Skipped 2 previous similar messages [12805.096520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.349758] Lustre: server umount lustre-MDT0000 complete [12810.515410] LustreError: 405325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759012 with bad export cookie 10064826306991719338 [12810.515913] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12810.532645] LustreError: 405325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12810.856656] Lustre: server umount lustre-MDT0001 complete [12822.869809] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12824.276689] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 21:57:06 (1786759026) [12830.958552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12831.362395] LustreError: 407786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12831.391313] LustreError: 407786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12831.469303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12836.080276] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12836.835064] LustreError: 407787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.933983] LustreError: 407786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12844.884269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12845.211761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12850.822419] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12861.743869] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12870.166362] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12877.581330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12877.960503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12885.446487] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [12893.651648] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12895.596084] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12903.884544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12911.080507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12911.090502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.107772] Lustre: Skipped 1 previous similar message [12911.122883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12911.126883] Lustre: Skipped 1 previous similar message [12912.104696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12915.861603] Lustre: server umount lustre-OST0000 complete [12921.842398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12921.856517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.878993] Lustre: Skipped 1 previous similar message [12921.902781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.912435] Lustre: Skipped 2 previous similar messages [12932.580143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12932.584262] Lustre: Skipped 5 previous similar messages [12935.137068] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12935.486882] Lustre: server umount lustre-MDT0000 complete [12937.704961] LustreError: 407787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12937.735768] LustreError: 407787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12939.634308] LustreError: 407766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759141 with bad export cookie 10064826306991720213 [12939.642308] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12939.647795] LustreError: 407766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12939.903588] Lustre: server umount lustre-MDT0001 complete [12951.469332] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12955.583489] Key type lgssc unregistered [12955.976677] LNet: 411519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12955.995318] LNetError: 411519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12956.008682] LNet: Removed LNI 192.168.202.137@tcp [12957.068375] Key type .llcrypt unregistered [12957.072233] Key type ._llcrypt unregistered [12989.640269] Key type ._llcrypt registered [12989.643869] Key type .llcrypt registered [12989.738773] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [12989.965534] Key type .llcrypt unregistered [12989.967121] Key type ._llcrypt unregistered [13009.780838] Key type ._llcrypt registered [13009.788740] Key type .llcrypt registered [13009.876292] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13011.134660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13011.156798] alg: No test for adler32 (adler32-zlib) [13012.166142] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13012.471676] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13014.248268] Key type lgssc registered [13015.321648] Lustre: Echo OBD driver; http://www.lustre.org/ [13023.846156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13023.869903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.436025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13029.992429] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13034.979092] LustreError: 413742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13035.012394] LustreError: 413742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13039.142156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13039.454571] LustreError: 413743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13039.485668] LustreError: 413743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13039.501372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13044.720433] LustreError: 414486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.742172] LustreError: 414486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13046.938654] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13049.826896] LustreError: 413739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.855166] LustreError: 413739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13050.871643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13055.775565] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13055.977952] LustreError: 413736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.999241] LustreError: 413736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13063.987840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13064.450105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13070.340852] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13081.569431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13081.572494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.613633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13085.157647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.164362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13085.173899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13087.595420] Lustre: server umount lustre-OST0000 complete [13095.116828] Lustre: server umount lustre-MDT0000 complete [13095.396104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13095.417207] Lustre: Skipped 1 previous similar message [13095.427803] LustreError: 413739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13095.454148] LustreError: 413739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13099.167140] LustreError: 413724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759301 with bad export cookie 558816180318359026 [13099.168070] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13099.192033] LustreError: 413724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13099.783391] Lustre: server umount lustre-MDT0001 complete [13110.835552] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13113.538956] Key type lgssc unregistered [13113.793976] LNet: 416905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13113.798095] LNetError: 416905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13113.810730] LNet: Removed LNI 192.168.202.137@tcp [13114.695830] Key type .llcrypt unregistered [13114.698527] Key type ._llcrypt unregistered [13145.026439] Key type ._llcrypt registered [13145.029708] Key type .llcrypt registered [13145.245484] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13147.404745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13147.484628] alg: No test for adler32 (adler32-zlib) [13148.893056] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13149.339942] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13151.193058] Key type lgssc registered [13153.162114] Lustre: Echo OBD driver; http://www.lustre.org/ [13162.832878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13162.877097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13178.528503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13183.648340] LustreError: 418295:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.137@tcp: failed processing log, type 4: rc = -110 [13210.336198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13217.018663] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13225.910536] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13232.757396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13232.831806] LustreError: 417631:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ef5f8da8 x1873552973236480/t0(0) o250->MGC192.168.202.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13233.148801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13233.203302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13238.017520] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13238.244456] LustreError: 418320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.272080] LustreError: 418320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13243.367434] LustreError: 418321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.417262] LustreError: 419332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.435394] LustreError: 419332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13246.252286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13246.532606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13251.588673] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13263.197694] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13271.714672] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13279.206650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13279.218483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13279.245512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13281.149379] Lustre: server umount lustre-OST0000 complete [13286.889436] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13286.893846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13286.908600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13287.319554] Lustre: server umount lustre-MDT0000 complete [13287.396951] LustreError: 419337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13291.118250] LustreError: 418302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759493 with bad export cookie 360024450363630345 [13291.149887] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13291.152509] LustreError: 418302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13301.867819] Lustre: server umount lustre-MDT0001 complete [13314.348820] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13317.819291] Key type lgssc unregistered [13318.187183] LNet: 421870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13318.194769] LNetError: 421870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13318.231530] LNet: Removed LNI 192.168.202.137@tcp [13319.127607] Key type .llcrypt unregistered [13319.131861] Key type ._llcrypt unregistered [13353.482970] Key type ._llcrypt registered [13353.484528] Key type .llcrypt registered [13353.547508] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13354.563661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13354.585751] alg: No test for adler32 (adler32-zlib) [13355.516421] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13355.710959] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13357.353193] Key type lgssc registered [13358.465047] Lustre: Echo OBD driver; http://www.lustre.org/ [13368.186759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13368.223408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13369.812719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13375.502987] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13379.046347] LustreError: 423291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.062926] LustreError: 423291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13384.166303] LustreError: 423292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13385.488847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13386.177693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13391.750793] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13400.247342] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13406.961831] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13415.269736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13415.922226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13425.213360] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13434.202966] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13439.479368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13444.587272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.590304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13444.606628] Lustre: Skipped 2 previous similar messages [13444.618964] Lustre: Skipped 2 previous similar messages [13446.770690] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.37@tcp (stopping) [13449.704234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13449.712699] Lustre: Skipped 2 previous similar messages [13450.358116] Lustre: server umount lustre-MDT0000 complete [13451.859433] LustreError: 423287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13453.129500] LustreError: 423270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.37@tcp arrived at 1786759655 with bad export cookie 11720275949525888311 [13453.132069] LustreError: 423270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13454.823145] LustreError: 423291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.852412] LustreError: 423291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13454.971678] LustreError: 423271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759657 with bad export cookie 11720275949525887646 [13454.991371] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13456.869710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13456.875789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13456.886457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13459.940500] LustreError: 423292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.958297] LustreError: 423292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13461.196740] Lustre: server umount lustre-OST0000 complete [13469.734599] Lustre: server umount lustre-MDT0001 complete [13482.732587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13486.461754] Key type lgssc unregistered [13487.053134] LNet: 426981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13487.066428] LNetError: 426981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13487.099059] LNet: Removed LNI 192.168.202.137@tcp [13488.254953] Key type .llcrypt unregistered [13488.259295] Key type ._llcrypt unregistered [13519.413599] Key type ._llcrypt registered [13519.419759] Key type .llcrypt registered [13519.524786] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13521.392502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13521.507455] alg: No test for adler32 (adler32-zlib) [13522.815471] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13523.172527] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13524.944249] Key type lgssc registered [13526.793750] Lustre: Echo OBD driver; http://www.lustre.org/ [13537.464524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13537.498920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13539.486629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13544.067644] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13549.042029] LustreError: 428403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.064410] LustreError: 428403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13554.108713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13554.152958] LustreError: 428404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13554.598196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13559.227094] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13567.364571] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13573.644416] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13581.178568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13581.624538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13585.734290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13588.871550] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13596.929533] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13600.739693] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13600.747851] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13600.767794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13603.925424] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.37@tcp (stopping) [13605.349410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13605.355808] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13607.327929] Lustre: server umount lustre-MDT0001 complete [13609.052635] LustreError: 429769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13615.588466] LustreError: 428403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13615.589262] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13615.621635] LustreError: 428403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13615.660349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13615.664570] Lustre: Skipped 1 previous similar message [13620.708595] LustreError: 428404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13620.710701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13621.188490] Lustre: server umount lustre-OST0000 complete [13626.688433] Lustre: server umount lustre-MDT0000 complete [13642.328511] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13645.698675] Key type lgssc unregistered [13646.008842] LNet: 432093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13646.021983] LNetError: 432093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13646.045290] LNet: Removed LNI 192.168.202.137@tcp [13647.034336] Key type .llcrypt unregistered [13647.046322] Key type ._llcrypt unregistered [13677.418898] Key type ._llcrypt registered [13677.424180] Key type .llcrypt registered [13677.524901] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [13692.386342] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13693.610269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13693.648104] alg: No test for adler32 (adler32-zlib) [13694.831047] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13695.057648] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13696.736293] Key type lgssc registered [13697.686691] Lustre: Echo OBD driver; http://www.lustre.org/ [13737.694956] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13754.017452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13754.039354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13755.337611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13755.368110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13755.479722] Lustre: lustre-MDT0000: new disk, initializing [13755.544789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13755.564498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13759.695369] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13775.220324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13775.670293] Lustre: lustre-OST0000: new disk, initializing [13775.674516] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13775.679885] Lustre: Skipped 1 previous similar message [13775.686319] Lustre: 437142:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13775.799105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13782.182148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13782.217262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13782.281358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13787.254185] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13798.290677] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13818.906862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.997443] Lustre: 436123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13819.052833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13819.109536] Lustre: lustre-MDT0001: new disk, initializing [13819.214738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13819.245492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13819.256368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13825.445973] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [13837.874693] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13838.162505] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13843.112713] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13843.455273] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13852.648525] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13852.653312] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13852.664487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13852.680839] Lustre: Skipped 1 previous similar message [13854.950782] Lustre: server umount lustre-MDT0001 complete [13862.894970] LustreError: 436134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13862.921547] LustreError: 436134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13868.001796] LustreError: 436131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13871.090487] LustreError: 436134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13871.125618] LustreError: 436134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13874.658170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13874.662907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13874.672990] Lustre: Skipped 1 previous similar message [13874.679150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13874.686309] Lustre: Skipped 1 previous similar message [13876.195886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13876.197720] LustreError: 436135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13878.355043] Lustre: server umount lustre-OST0000 complete [13881.316445] LustreError: 436134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13889.323458] Lustre: server umount lustre-MDT0000 complete [13907.849463] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13911.833612] Key type lgssc unregistered [13912.206673] LNet: 440895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13912.219247] LNetError: 440895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13912.236489] LNet: Removed LNI 192.168.202.137@tcp [13913.241463] Key type .llcrypt unregistered [13913.244479] Key type ._llcrypt unregistered [13958.805562] Key type ._llcrypt registered [13958.816508] Key type .llcrypt registered [13958.964474] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [13959.462618] Key type .llcrypt unregistered [13959.466525] Key type ._llcrypt unregistered [13977.648658] Key type ._llcrypt registered [13977.651526] Key type .llcrypt registered [13977.801403] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [13979.832463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13979.882992] alg: No test for adler32 (adler32-zlib) [13981.365540] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [13981.849948] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [13983.696247] Key type lgssc registered [13984.824790] Lustre: Echo OBD driver; http://www.lustre.org/ [14021.572152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14021.597697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14023.109625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14023.218254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14023.359107] Lustre: lustre-MDT0000: new disk, initializing [14023.510758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14023.530691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14029.528468] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14041.063823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14041.220531] Lustre: 444281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14041.329242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14041.341897] Lustre: Skipped 1 previous similar message [14041.427510] Lustre: lustre-MDT0001: new disk, initializing [14041.542812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14041.591906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14041.607758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14046.433560] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14056.262118] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14062.955353] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14070.238912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14070.686132] Lustre: lustre-OST0000: new disk, initializing [14070.689327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14070.694852] Lustre: 446249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14070.796250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14077.130779] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14079.081301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14079.101186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14079.188635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14086.797463] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14092.788926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14092.794561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14092.807909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14094.818879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14094.825642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14094.839028] Lustre: Skipped 1 previous similar message [14099.135462] Lustre: server umount lustre-MDT0000 complete [14103.239645] LustreError: 444274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760305 with bad export cookie 8767567833257787449 [14103.255940] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14103.630021] Lustre: server umount lustre-MDT0001 complete [14118.179616] Lustre: server umount lustre-OST0000 complete [14131.036599] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:18:52 (1786760332) [14143.360326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14144.500659] LustreError: 448279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14144.519885] LustreError: 448279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14144.660177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14150.123801] LustreError: 448280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14150.843936] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14154.212124] LustreError: 448279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14159.335744] LustreError: 448280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14162.816613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14163.335799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14168.977891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14180.860795] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14187.746526] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14209.649928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14210.018825] Lustre: Found index 0 for lustre-OST0000, updating log [14210.285278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14214.407767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14218.133654] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14227.296847] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14234.606470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14234.632440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14234.646185] Lustre: Skipped 2 previous similar messages [14240.358211] Lustre: server umount lustre-OST0000 complete [14260.705824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14260.713163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14260.721123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14260.723793] Lustre: Skipped 2 previous similar messages [14264.676243] Lustre: server umount lustre-MDT0000 complete [14267.884153] LustreError: 448276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14267.936088] LustreError: 448276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14269.324751] LustreError: 448262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760471 with bad export cookie 8767567833257788842 [14269.327209] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14269.348224] LustreError: 448262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [14270.416869] Lustre: server umount lustre-MDT0001 complete [14270.627953] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [14281.704327] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:21:23 (1786760483) [14912.654679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14913.247966] LustreError: 454695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14913.378161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14918.629441] LustreError: 454696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14918.848164] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14923.750531] LustreError: 454695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14928.742638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14929.171203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14933.976587] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14943.133206] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14950.732712] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14972.993921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14973.313664] Lustre: Found index 0 for lustre-OST0000, updating log [14973.717936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14974.124134] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [14980.715709] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [14988.722208] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14991.918972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14991.950186] Lustre: Skipped 1 previous similar message [14991.959955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14991.969186] Lustre: Skipped 4 previous similar messages [15005.152239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15005.261604] Lustre: server umount lustre-OST0000 complete [15011.042971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15011.069397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15011.079927] Lustre: Skipped 1 previous similar message [15012.132881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15014.187404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15014.199933] Lustre: Skipped 3 previous similar messages [15016.400904] Lustre: server umount lustre-MDT0000 complete [15017.265571] LustreError: 454696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15017.285898] LustreError: 454696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15020.360608] LustreError: 456546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761222 with bad export cookie 8767567833257789913 [15020.370237] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15020.374403] LustreError: 456546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15020.808973] Lustre: server umount lustre-MDT0001 complete [15034.528411] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp [15057.192863] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [15069.208846] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15115.126597] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:35:16 (1786761316) [15136.397167] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15140.155851] Key type lgssc unregistered [15140.480283] LNet: 461314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15140.497323] LNetError: 461314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15234.733562] LNet: Removed LNI 192.168.202.137@tcp [15235.975696] Key type .llcrypt unregistered [15235.977154] Key type ._llcrypt unregistered [15254.900877] Key type ._llcrypt registered [15254.902617] Key type .llcrypt registered [15255.087659] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15256.699123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15256.916468] alg: No test for adler32 (adler32-zlib) [15258.163561] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [15258.466254] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15260.184206] Key type lgssc registered [15261.599304] Lustre: Echo OBD driver; http://www.lustre.org/ [15266.988980] LNet: Added LNI 192.168.202.137@tcp100 [8/256/0/180] [15268.128775] LNet: Added LNI 192.168.202.137@tcp101 [8/256/0/180] [15269.091224] LNet: Added LNI 192.168.202.137@tcp102 [8/256/0/180] [15270.269939] LNet: Added LNI 192.168.202.137@tcp103 [8/256/0/180] [15271.420372] LNet: Added LNI 192.168.202.137@tcp104 [8/256/0/180] [15272.462235] LNet: Added LNI 192.168.202.137@tcp105 [8/256/0/180] [15273.495765] LNet: Added LNI 192.168.202.137@tcp106 [8/256/0/180] [15274.448542] LNet: Added LNI 192.168.202.137@tcp107 [8/256/0/180] [15275.487168] LNet: Added LNI 192.168.202.137@tcp108 [8/256/0/180] [15276.381936] LNet: Added LNI 192.168.202.137@tcp109 [8/256/0/180] [15277.474648] LNet: Added LNI 192.168.202.137@tcp110 [8/256/0/180] [15278.680505] LNet: Added LNI 192.168.202.137@tcp111 [8/256/0/180] [15279.987913] LNet: Added LNI 192.168.202.137@tcp112 [8/256/0/180] [15280.993125] LNet: Added LNI 192.168.202.137@tcp113 [8/256/0/180] [15282.171322] LNet: Added LNI 192.168.202.137@tcp114 [8/256/0/180] [15283.306631] LNet: Added LNI 192.168.202.137@tcp115 [8/256/0/180] [15284.461542] LNet: Added LNI 192.168.202.137@tcp116 [8/256/0/180] [15285.491734] LNet: Added LNI 192.168.202.137@tcp117 [8/256/0/180] [15286.362463] LNet: Added LNI 192.168.202.137@tcp118 [8/256/0/180] [15287.246593] LNet: Added LNI 192.168.202.137@tcp119 [8/256/0/180] [15288.162351] LNet: Added LNI 192.168.202.137@tcp120 [8/256/0/180] [15289.152392] LNet: Added LNI 192.168.202.137@tcp121 [8/256/0/180] [15290.302849] LNet: Added LNI 192.168.202.137@tcp122 [8/256/0/180] [15291.491388] LNet: Added LNI 192.168.202.137@tcp123 [8/256/0/180] [15292.454889] LNet: Added LNI 192.168.202.137@tcp124 [8/256/0/180] [15293.268040] LNet: Added LNI 192.168.202.137@tcp125 [8/256/0/180] [15294.143442] LNet: Added LNI 192.168.202.137@tcp126 [8/256/0/180] [15295.015472] LNet: Added LNI 192.168.202.137@tcp127 [8/256/0/180] [15296.060811] LNet: Added LNI 192.168.202.137@tcp128 [8/256/0/180] [15297.212702] LNet: Added LNI 192.168.202.137@tcp129 [8/256/0/180] [15298.252706] LNet: Added LNI 192.168.202.137@tcp130 [8/256/0/180] [15299.366153] LNet: Added LNI 192.168.202.137@tcp131 [8/256/0/180] [15300.533992] LNet: Added LNI 192.168.202.137@tcp132 [8/256/0/180] [15301.751962] LNet: Added LNI 192.168.202.137@tcp133 [8/256/0/180] [15302.812737] LNet: Added LNI 192.168.202.137@tcp134 [8/256/0/180] [15303.966871] LNet: Added LNI 192.168.202.137@tcp135 [8/256/0/180] [15311.718901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15311.738156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15312.934172] Lustre: 464343:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15313.005253] LNetError: 464365:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15313.038170] LNetError: Protocol error connecting to 192.168.202.137@tcp120 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15313.050444] LNetError: Skipped 4 previous similar messages [15313.224631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15313.284160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15313.366926] Lustre: lustre-MDT0000: new disk, initializing [15313.489472] Lustre: 464401:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15313.499544] Lustre: 464401:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15313.526652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15313.543899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15313.842025] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp129 [15314.083840] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15314.104898] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15314.138891] LNetError: Protocol error connecting to 192.168.202.137@tcp122 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15314.162325] LNetError: Skipped 3 previous similar messages [15316.902387] LNetError: 464364:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15316.910127] LNetError: 464364:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [15316.916819] LNetError: Protocol error connecting to 192.168.202.137@tcp129 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15316.925198] LNetError: Skipped 1 previous similar message [15319.766614] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15320.998140] LNetError: 461802:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15321.012847] LNetError: Protocol error connecting to 192.168.202.137@tcp129 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15331.377582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15331.555560] Lustre: 464361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15331.617903] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15331.622531] Lustre: Skipped 1 previous similar message [15331.689233] Lustre: lustre-MDT0001: new disk, initializing [15331.714564] Lustre: 465164:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15331.722574] Lustre: 465164:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15331.882582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15331.963528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15332.002629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15336.641976] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15339.065087] Lustre: 465529:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15339.081896] Lustre: 465529:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15342.443698] LNetError: 464373:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15342.453173] LNetError: Protocol error connecting to 192.168.202.137@tcp at host 192.168.202.137:988: is it running a compatible version of Lustre? [15347.774812] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15351.656287] LNetError: 464364:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15351.663784] LNetError: 464364:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15351.677051] LNetError: Protocol error connecting to 192.168.202.137@tcp109 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15351.705755] LNetError: Skipped 6 previous similar messages [15352.835843] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15360.317211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15360.677974] Lustre: lustre-OST0000: new disk, initializing [15360.684159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15360.691860] Lustre: 466337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15360.846353] Lustre: 466342:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.137@tcp: can't add NID 192.168.202.137@tcp130: rc = -75 [15360.851776] Lustre: 466342:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15360.944347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15367.284144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15367.291802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15367.441626] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15367.469992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15368.098767] LNetError: 461803:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15368.106439] LNetError: 461803:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [15368.112424] LNetError: Protocol error connecting to 192.168.202.137@tcp123 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15368.130589] LNetError: Skipped 12 previous similar messages [15373.216297] LNet: 1 local NIs in recovery (showing 1): 192.168.202.137@tcp129 [15377.666963] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15388.141415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15388.155521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15388.169563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15388.180806] Lustre: Skipped 1 previous similar message [15388.174209] LustreError: Skipped 1 previous similar message [15391.785878] Lustre: server umount lustre-OST0000 complete [15398.374941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15398.391938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15398.399854] Lustre: Skipped 1 previous similar message [15398.401614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15398.419548] Lustre: Skipped 1 previous similar message [15400.865963] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.137 [15400.881757] LNetError: 464366:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [15400.896501] LNetError: Protocol error connecting to 192.168.202.137@tcp106 at host 192.168.202.137:988: is it running a compatible version of Lustre? [15400.920136] LNetError: Skipped 15 previous similar messages [15403.490974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15403.498034] Lustre: Skipped 1 previous similar message [15408.614817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15408.633578] Lustre: Skipped 1 previous similar message [15411.168151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15411.331390] Lustre: server umount lustre-MDT0000 complete [15416.908407] LustreError: 464354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761619 with bad export cookie 16504886144927027429 [15416.911892] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15416.940984] LustreError: 464354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15417.547960] Lustre: server umount lustre-MDT0001 complete [15431.798346] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15435.069468] Key type lgssc unregistered [15435.580569] LNet: 468256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15435.609527] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15436.641973] LNet: Removed LNI 192.168.202.137@tcp [15436.652662] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15436.673534] LNet: Removed LNI 192.168.202.137@tcp100 [15437.728167] LNet: Removed LNI 192.168.202.137@tcp101 [15437.730235] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15437.746943] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15437.766428] LNet: Removed LNI 192.168.202.137@tcp102 [15438.817532] LNet: Removed LNI 192.168.202.137@tcp103 [15439.840185] LNet: Removed LNI 192.168.202.137@tcp104 [15439.849691] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15439.869499] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15439.881996] LNet: Removed LNI 192.168.202.137@tcp105 [15440.928184] LNet: Removed LNI 192.168.202.137@tcp106 [15441.952735] LNet: Removed LNI 192.168.202.137@tcp107 [15442.976938] LNet: Removed LNI 192.168.202.137@tcp108 [15444.000169] LNet: Removed LNI 192.168.202.137@tcp109 [15444.035067] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15444.044931] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15444.051388] LNet: Removed LNI 192.168.202.137@tcp110 [15445.088189] LNet: Removed LNI 192.168.202.137@tcp111 [15446.112188] LNet: Removed LNI 192.168.202.137@tcp112 [15447.136214] LNet: Removed LNI 192.168.202.137@tcp113 [15448.160180] LNet: Removed LNI 192.168.202.137@tcp114 [15449.184216] LNet: Removed LNI 192.168.202.137@tcp115 [15450.208518] LNet: Removed LNI 192.168.202.137@tcp116 [15451.234351] LNet: Removed LNI 192.168.202.137@tcp117 [15452.320459] LNet: Removed LNI 192.168.202.137@tcp118 [15452.357354] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15452.375852] LNetError: 468256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15452.397712] LNet: Removed LNI 192.168.202.137@tcp119 [15453.473720] LNet: Removed LNI 192.168.202.137@tcp120 [15454.496327] LNet: Removed LNI 192.168.202.137@tcp121 [15455.522240] LNet: Removed LNI 192.168.202.137@tcp122 [15456.545123] LNet: Removed LNI 192.168.202.137@tcp123 [15457.568309] LNet: Removed LNI 192.168.202.137@tcp124 [15458.592332] LNet: Removed LNI 192.168.202.137@tcp125 [15459.616307] LNet: Removed LNI 192.168.202.137@tcp126 [15460.640282] LNet: Removed LNI 192.168.202.137@tcp127 [15461.664331] LNet: Removed LNI 192.168.202.137@tcp128 [15461.671859] LNet: Removed LNI 192.168.202.137@tcp129 [15461.676597] LNet: Removed LNI 192.168.202.137@tcp130 [15461.684362] LNet: Removed LNI 192.168.202.137@tcp131 [15461.690264] LNet: Removed LNI 192.168.202.137@tcp132 [15461.696424] LNet: Removed LNI 192.168.202.137@tcp133 [15461.703334] LNet: Removed LNI 192.168.202.137@tcp134 [15461.732479] LNet: Removed LNI 192.168.202.137@tcp135 [15462.637098] Key type .llcrypt unregistered [15462.639754] Key type ._llcrypt unregistered [15488.360727] Key type ._llcrypt registered [15488.365412] Key type .llcrypt registered [15488.542768] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [15504.878210] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15506.202363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15506.271914] alg: No test for adler32 (adler32-zlib) [15507.359912] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [15507.582970] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15509.296212] Key type lgssc registered [15510.767844] Lustre: Echo OBD driver; http://www.lustre.org/ [15547.372982] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:42:29 (1786761749) [15565.159529] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [15567.939655] Key type lgssc unregistered [15568.273563] LNet: 472344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15568.285994] LNetError: 472344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15568.319289] LNet: Removed LNI 192.168.202.137@tcp [15569.388539] Key type .llcrypt unregistered [15569.390806] Key type ._llcrypt unregistered [15587.272677] Key type ._llcrypt registered [15587.276463] Key type .llcrypt registered [15587.496965] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [15589.356436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15589.422652] alg: No test for adler32 (adler32-zlib) [15590.489466] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [15590.697599] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [15592.385723] Key type lgssc registered [15594.072809] Lustre: Echo OBD driver; http://www.lustre.org/ [15606.003376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15606.042571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15607.411451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15607.442939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15607.592698] Lustre: lustre-MDT0000: new disk, initializing [15607.657057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15607.671938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15612.416891] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15625.261647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15625.491707] Lustre: 473559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15625.611299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15625.617526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15625.622029] Lustre: Skipped 1 previous similar message [15625.727210] Lustre: lustre-MDT0001: new disk, initializing [15625.818727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15625.873295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15625.886177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15630.634931] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15641.464208] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15648.202587] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15655.772638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15656.097872] Lustre: lustre-OST0000: new disk, initializing [15656.104099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15656.116380] Lustre: 475528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15656.226971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15663.485725] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [15666.294229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15666.305691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15666.436099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15674.287479] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15678.793976] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing lnet_if_list [15685.697849] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15685.749174] LNet: Added LNI 192.168.202.137@tcp10 [8/256/0/180] [15693.777421] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15693.807590] LNet: Added LNI 192.168.202.137@tcp11 [8/256/0/180] [15698.699724] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15698.728695] LNet: Added LNI 192.168.202.137@tcp12 [8/256/0/180] [15704.593754] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15704.618278] LNet: Added LNI 192.168.202.137@tcp13 [8/256/0/180] [15709.087412] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15709.107375] LNet: Added LNI 192.168.202.137@tcp14 [8/256/0/180] [15714.460423] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15714.484795] LNet: Added LNI 192.168.202.137@tcp15 [8/256/0/180] [15720.593139] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15720.608618] LNet: Added LNI 192.168.202.137@tcp16 [8/256/0/180] [15726.212813] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15726.257661] LNet: Added LNI 192.168.202.137@tcp17 [8/256/0/180] [15731.033934] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15731.072885] LNet: Added LNI 192.168.202.137@tcp18 [8/256/0/180] [15736.776496] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15736.801360] LNet: Added LNI 192.168.202.137@tcp19 [8/256/0/180] [15743.365600] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15743.397604] LNet: Added LNI 192.168.202.137@tcp20 [8/256/0/180] [15750.322664] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15750.377457] LNet: Added LNI 192.168.202.137@tcp21 [8/256/0/180] [15756.889077] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15756.928216] LNet: Added LNI 192.168.202.137@tcp22 [8/256/0/180] [15763.294065] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15763.350814] LNet: Added LNI 192.168.202.137@tcp23 [8/256/0/180] [15769.951373] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15769.978634] LNet: Added LNI 192.168.202.137@tcp24 [8/256/0/180] [15777.220995] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15777.287277] LNet: Added LNI 192.168.202.137@tcp25 [8/256/0/180] [15786.658968] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15786.711473] LNet: Added LNI 192.168.202.137@tcp26 [8/256/0/180] [15794.078597] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15794.118467] LNet: Added LNI 192.168.202.137@tcp27 [8/256/0/180] [15801.442625] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15801.525486] LNet: Added LNI 192.168.202.137@tcp28 [8/256/0/180] [15808.986884] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15809.040183] LNet: Added LNI 192.168.202.137@tcp29 [8/256/0/180] [15816.509390] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15816.553503] LNet: Added LNI 192.168.202.137@tcp30 [8/256/0/180] [15823.587734] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15823.619752] LNet: Added LNI 192.168.202.137@tcp31 [8/256/0/180] [15834.481970] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15834.585901] LNet: Added LNI 192.168.202.137@tcp32 [8/256/0/180] [15841.613404] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15841.665517] LNet: Added LNI 192.168.202.137@tcp33 [8/256/0/180] [15849.116861] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15849.180035] LNet: Added LNI 192.168.202.137@tcp34 [8/256/0/180] [15857.218528] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15857.250221] LNet: Added LNI 192.168.202.137@tcp35 [8/256/0/180] [15864.252824] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15864.280187] LNet: Added LNI 192.168.202.137@tcp36 [8/256/0/180] [15871.007782] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15871.044135] LNet: Added LNI 192.168.202.137@tcp37 [8/256/0/180] [15878.588584] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15878.646718] LNet: Added LNI 192.168.202.137@tcp38 [8/256/0/180] [15886.473987] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15886.495915] LNet: Added LNI 192.168.202.137@tcp39 [8/256/0/180] [15895.098414] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15895.149373] LNet: Added LNI 192.168.202.137@tcp40 [8/256/0/180] [15904.353298] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15904.382811] LNet: Added LNI 192.168.202.137@tcp41 [8/256/0/180] [15912.494556] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15912.544472] LNet: Added LNI 192.168.202.137@tcp42 [8/256/0/180] [15921.025214] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15921.076508] LNet: Added LNI 192.168.202.137@tcp43 [8/256/0/180] [15928.552934] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15928.586485] LNet: Added LNI 192.168.202.137@tcp44 [8/256/0/180] [15936.687828] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15936.709097] LNet: Added LNI 192.168.202.137@tcp45 [8/256/0/180] [15944.531338] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15944.557765] LNet: Added LNI 192.168.202.137@tcp46 [8/256/0/180] [15954.640218] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15954.682730] LNet: Added LNI 192.168.202.137@tcp47 [8/256/0/180] [15965.240137] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15965.266503] LNet: Added LNI 192.168.202.137@tcp48 [8/256/0/180] [15974.472456] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15974.502689] LNet: Added LNI 192.168.202.137@tcp49 [8/256/0/180] [15984.181947] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15984.212675] LNet: Added LNI 192.168.202.137@tcp50 [8/256/0/180] [16004.580781] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16004.586592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16004.603199] Lustre: Skipped 2 previous similar messages [16004.603585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16004.611789] Lustre: Skipped 2 previous similar messages [16009.705288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16009.711968] Lustre: Skipped 2 previous similar messages [16010.108729] Lustre: server umount lustre-MDT0000 complete [16014.481353] LustreError: 473551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762216 with bad export cookie 4855434146154776043 [16014.489241] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16014.494859] LustreError: 473551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16014.817633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16015.058285] Lustre: server umount lustre-MDT0001 complete [16036.483567] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16046.892795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16060.314448] LustreError: 475522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16060.331116] LustreError: 475522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16060.433170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16065.139559] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16065.512858] LustreError: 475523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16070.629524] LustreError: 475522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16070.643378] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16074.664725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16074.954893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16080.106832] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16080.217535] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16089.227209] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16096.561952] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16121.316490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16121.332698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16121.336221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16121.358475] Lustre: Skipped 1 previous similar message [16123.910044] Lustre: server umount lustre-OST0000 complete [16131.557178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16131.562036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16131.564962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16131.564972] Lustre: Skipped 1 previous similar message [16131.597729] Lustre: Skipped 1 previous similar message [16135.058116] Lustre: server umount lustre-MDT0000 complete [16136.674591] LustreError: 499776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16136.688302] LustreError: 499776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16139.528623] LustreError: 473552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762341 with bad export cookie 4855434146154778731 [16139.538160] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16139.540146] LustreError: 473552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16140.372944] Lustre: server umount lustre-MDT0001 complete [16152.071298] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16156.210510] Key type lgssc unregistered [16157.045676] LNet: 502581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16157.057722] LNetError: 502581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16157.080794] LNet: Removed LNI 192.168.202.137@tcp [16157.083100] LNet: Removed LNI 192.168.202.137@tcp10 [16157.098301] LNet: Removed LNI 192.168.202.137@tcp11 [16157.104640] LNet: Removed LNI 192.168.202.137@tcp12 [16157.114426] LNet: Removed LNI 192.168.202.137@tcp13 [16157.123534] LNet: Removed LNI 192.168.202.137@tcp14 [16157.129623] LNet: Removed LNI 192.168.202.137@tcp15 [16157.144130] LNet: Removed LNI 192.168.202.137@tcp16 [16157.158018] LNet: Removed LNI 192.168.202.137@tcp17 [16157.166743] LNet: Removed LNI 192.168.202.137@tcp18 [16157.176212] LNet: Removed LNI 192.168.202.137@tcp19 [16157.187352] LNet: Removed LNI 192.168.202.137@tcp20 [16157.189249] LNet: Removed LNI 192.168.202.137@tcp21 [16157.197201] LNet: Removed LNI 192.168.202.137@tcp22 [16157.199138] LNet: Removed LNI 192.168.202.137@tcp23 [16157.204534] LNet: Removed LNI 192.168.202.137@tcp24 [16157.214338] LNet: Removed LNI 192.168.202.137@tcp25 [16157.228165] LNet: Removed LNI 192.168.202.137@tcp26 [16157.232170] LNet: Removed LNI 192.168.202.137@tcp27 [16157.235408] LNet: Removed LNI 192.168.202.137@tcp28 [16157.238698] LNet: Removed LNI 192.168.202.137@tcp29 [16157.240995] LNet: Removed LNI 192.168.202.137@tcp30 [16157.243529] LNet: Removed LNI 192.168.202.137@tcp31 [16157.251172] LNet: Removed LNI 192.168.202.137@tcp32 [16157.257961] LNet: Removed LNI 192.168.202.137@tcp33 [16157.263487] LNet: Removed LNI 192.168.202.137@tcp34 [16157.266466] LNet: Removed LNI 192.168.202.137@tcp35 [16157.268525] LNet: Removed LNI 192.168.202.137@tcp36 [16157.274453] LNet: Removed LNI 192.168.202.137@tcp37 [16157.283470] LNet: Removed LNI 192.168.202.137@tcp38 [16157.291658] LNet: Removed LNI 192.168.202.137@tcp39 [16157.298524] LNet: Removed LNI 192.168.202.137@tcp40 [16157.309929] LNet: Removed LNI 192.168.202.137@tcp41 [16157.322232] LNet: Removed LNI 192.168.202.137@tcp42 [16157.329190] LNet: Removed LNI 192.168.202.137@tcp43 [16157.341024] LNet: Removed LNI 192.168.202.137@tcp44 [16157.350397] LNet: Removed LNI 192.168.202.137@tcp45 [16157.355839] LNet: Removed LNI 192.168.202.137@tcp46 [16157.366469] LNet: Removed LNI 192.168.202.137@tcp47 [16157.371370] LNet: Removed LNI 192.168.202.137@tcp48 [16157.373982] LNet: Removed LNI 192.168.202.137@tcp49 [16157.382923] LNet: Removed LNI 192.168.202.137@tcp50 [16158.793187] Key type .llcrypt unregistered [16158.796901] Key type ._llcrypt unregistered [16182.432720] Key type ._llcrypt registered [16182.434205] Key type .llcrypt registered [16182.522524] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16199.813546] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16201.651144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16201.734441] alg: No test for adler32 (adler32-zlib) [16202.926406] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [16203.334545] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16205.072220] Key type lgssc registered [16207.194371] Lustre: Echo OBD driver; http://www.lustre.org/ [16245.300467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16245.342154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16246.538167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16246.557989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16246.607321] Lustre: lustre-MDT0000: new disk, initializing [16246.654676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16246.672401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16250.725401] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16261.628518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16261.791546] Lustre: 505781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16261.870139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16261.876326] Lustre: Skipped 1 previous similar message [16262.018788] Lustre: lustre-MDT0001: new disk, initializing [16262.199381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16262.271258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16262.305629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16268.696830] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16279.429433] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16284.747471] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16294.113715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16294.384507] Lustre: *** cfs_fail_loc=90f, val=0*** [16294.474585] Lustre: lustre-OST0000: new disk, initializing [16294.478586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16294.498459] Lustre: 507798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16294.669642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16301.712956] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16302.177599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16302.192061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16302.310076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16312.691786] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16323.046660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16323.050729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16323.056710] LustreError: Skipped 1 previous similar message [16323.061585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16323.070807] Lustre: Skipped 1 previous similar message [16323.086874] Lustre: Skipped 1 previous similar message [16327.028126] Lustre: server umount lustre-OST0000 complete [16333.281193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16333.307543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16333.318535] Lustre: Skipped 1 previous similar message [16338.031921] Lustre: server umount lustre-MDT0000 complete [16342.308498] LustreError: 505773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762544 with bad export cookie 13302217322474289360 [16342.313574] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16342.322915] LustreError: 505773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16342.662392] Lustre: server umount lustre-MDT0001 complete [16354.353388] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16357.621294] Key type lgssc unregistered [16357.941765] LNet: 509708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16357.947904] LNetError: 509708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16357.967369] LNet: Removed LNI 192.168.202.137@tcp [16358.910782] Key type .llcrypt unregistered [16358.913141] Key type ._llcrypt unregistered [16384.225961] Key type ._llcrypt registered [16384.228170] Key type .llcrypt registered [16384.313702] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16385.547481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16385.572232] alg: No test for adler32 (adler32-zlib) [16386.532987] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [16386.880435] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16388.608245] Key type lgssc registered [16389.824011] Lustre: Echo OBD driver; http://www.lustre.org/ [16397.934722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16397.955797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16399.486657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16404.697647] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16409.062711] LustreError: 510992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16409.079608] LustreError: 510992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16414.186657] LustreError: 510993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16415.370279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16415.670764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16420.168380] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16428.355635] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16434.231558] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16441.172746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16441.531407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16450.733558] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16460.643208] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16471.542449] loop0: detected capacity change from 0 to 204800000 [16471.750281] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16471.758444] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16471.780900] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16478.261046] loop0: detected capacity change from 0 to 204800000 [16478.561209] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16478.585551] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16478.601197] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16485.420887] loop0: detected capacity change from 0 to 204800000 [16491.891225] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16492.040182] Lustre: fs15246-OST0000: new disk, initializing [16492.045695] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16492.049433] Lustre: 514587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16492.087859] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16499.731284] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16511.042654] loop1: detected capacity change from 0 to 204800000 [16516.182652] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16516.320124] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16516.416970] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16516.563881] Lustre: fs15246-MDT0000: new disk, initializing [16516.859256] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16516.899503] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16517.068192] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16518.116292] Lustre: 515711:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000047bbc00d x1873556367524096/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786762736 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16521.342559] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16523.176539] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16523.182684] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16523.225889] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16547.826588] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16547.845595] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16552.075233] Lustre: server umount fs15246-MDT0000 complete [16552.951260] LustreError: 510987:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16558.055045] LustreError: 516133:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16563.186980] LustreError: 510988:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16563.705940] Lustre: server umount fs15246-OST0000 complete [16583.142850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16583.152163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16583.156295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16583.193796] Lustre: Skipped 1 previous similar message [16583.208746] Lustre: Skipped 1 previous similar message [16588.799302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16588.814557] Lustre: Skipped 1 previous similar message [16592.352176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16592.734342] Lustre: server umount lustre-OST0000 complete [16599.013704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16599.032164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16599.043783] Lustre: Skipped 1 previous similar message [16604.032231] Lustre: server umount lustre-MDT0000 complete [16604.132449] LustreError: 510993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16604.155976] LustreError: 510993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16608.250190] LustreError: 510974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762810 with bad export cookie 15981869508402487504 [16608.262641] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16608.267907] LustreError: 510974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16608.580476] Lustre: server umount lustre-MDT0001 complete [16623.080697] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [16626.942913] Key type lgssc unregistered [16627.377425] LNet: 518814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16627.399053] LNetError: 518814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16627.419820] LNet: Removed LNI 192.168.202.137@tcp [16628.692288] Key type .llcrypt unregistered [16628.693905] Key type ._llcrypt unregistered [16650.351241] Key type ._llcrypt registered [16650.352991] Key type .llcrypt registered [16650.498736] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16652.428965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16652.472316] alg: No test for adler32 (adler32-zlib) [16653.648439] Lustre: Lustre: Build Version: 2.17.57_3_g6085841 [16653.894930] LNet: Added LNI 192.168.202.137@tcp [8/256/0/180] [16655.592368] Key type lgssc registered [16656.848460] Lustre: Echo OBD driver; http://www.lustre.org/ [16700.040249] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16710.403872] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16752.624295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16752.666541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16754.115398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16754.157054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16754.302758] Lustre: lustre-MDT0000: new disk, initializing [16754.423304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16754.457562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16759.517598] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16771.170823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16771.320232] Lustre: 523298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16771.380583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16771.384181] Lustre: Skipped 1 previous similar message [16771.467610] Lustre: lustre-MDT0001: new disk, initializing [16771.483908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16771.606841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16771.706346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16771.731663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16777.069261] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16787.309839] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16793.161223] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16801.782608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16802.243603] Lustre: lustre-OST0000: new disk, initializing [16802.250420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16802.259821] Lustre: 525265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16802.382475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16809.024292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16809.040062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16809.178048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16809.705333] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_default_debug -1 all [16821.666363] Lustre: DEBUG MARKER: oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16828.376710] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16828.669436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16834.266327] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16834.637704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16840.165931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16840.173907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16840.181564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16840.191745] LustreError: Skipped 1 previous similar message [16840.241810] Lustre: Skipped 1 previous similar message [16843.563366] Lustre: server umount lustre-OST0000 complete [16850.420123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16850.442070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16850.454114] Lustre: Skipped 1 previous similar message [16850.463918] Lustre: Skipped 1 previous similar message [16854.959462] Lustre: server umount lustre-MDT0000 complete [16859.566228] LustreError: 524093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763061 with bad export cookie 1653839352161928573 [16859.578450] LustreError: MGC192.168.202.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16860.196706] Lustre: server umount lustre-MDT0001 complete [16892.148548] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing set_hostid [16907.111417] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing load_modules_local [16947.130745] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16614 sec ========= 23:05:48 (1786763148) [16949.258772] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:05:50 (1786763150) === [16953.105688] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:05:54 (1786763154) === [17006.513272] Lustre: DEBUG MARKER: oleg237-server.virtnet: executing unload_modules_local [17010.131397] Key type lgssc unregistered [17010.494756] LNet: 531933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17010.502836] LNetError: 531933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17011.567704] LNet: Removed LNI 192.168.202.137@tcp [17012.819451] Key type .llcrypt unregistered [17012.821797] Key type ._llcrypt unregistered