[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 582978452 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002326] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009113] LSM: Security Framework initializing [ 0.010040] Yama: becoming mindful. [ 0.011026] SELinux: Initializing. [ 0.012052] *** VALIDATE selinux *** [ 0.020005] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025347] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028038] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029094] *** VALIDATE tmpfs *** [ 0.030415] *** VALIDATE proc *** [ 0.032092] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034237] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.040675] debug: unmapping init [mem 0xffffffff84459000-0xffffffff84460fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043802] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049059] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051284] rcu: Hierarchical SRCU implementation. [ 0.053656] smp: Bringing up secondary CPUs ... [ 0.054642] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.066135] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107490] node 0 deferred pages initialised in 36ms [ 0.111614] devtmpfs: initialized [ 0.112234] x86/mm: Memory block size: 128MB [ 0.115251] gcov: version magic: 0x41383552 [ 0.117294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120546] pinctrl core: initialized pinctrl subsystem [ 0.123214] [ 0.123900] ************************************************************* [ 0.126074] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129020] ** ** [ 0.132015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.135014] ** ** [ 0.138013] ** This means that this kernel is built to expose internal ** [ 0.141011] ** IOMMU data structures, which may compromise security on ** [ 0.143079] ** your system. ** [ 0.145053] ** ** [ 0.148014] ** If you see this message and you are not debugging the ** [ 0.151013] ** kernel, report this immediately to your vendor! ** [ 0.154013] ** ** [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ************************************************************* [ 0.163922] NET: Registered protocol family 16 [ 0.165453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176736] cpuidle: using governor menu [ 0.179012] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183033] PCI: Using configuration type 1 for base access [ 0.185127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190383] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193209] cryptd: max_cpu_qlen set to 1000 [ 0.195434] ACPI: Added _OSI(Module Device) [ 0.197017] ACPI: Added _OSI(Processor Device) [ 0.199013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206751] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215107] ACPI: Interpreter enabled [ 0.217058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219062] ACPI: Using IOAPIC for interrupt routing [ 0.221103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226549] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240448] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255735] acpiphp: Slot [2] registered [ 0.258194] acpiphp: Slot [5] registered [ 0.260125] acpiphp: Slot [6] registered [ 0.262112] acpiphp: Slot [7] registered [ 0.263180] acpiphp: Slot [8] registered [ 0.265165] acpiphp: Slot [9] registered [ 0.267170] acpiphp: Slot [10] registered [ 0.269168] acpiphp: Slot [3] registered [ 0.270111] acpiphp: Slot [4] registered [ 0.271101] acpiphp: Slot [11] registered [ 0.273107] acpiphp: Slot [12] registered [ 0.275199] acpiphp: Slot [13] registered [ 0.277134] acpiphp: Slot [14] registered [ 0.279115] acpiphp: Slot [15] registered [ 0.281123] acpiphp: Slot [16] registered [ 0.283143] acpiphp: Slot [17] registered [ 0.285117] acpiphp: Slot [18] registered [ 0.287118] acpiphp: Slot [19] registered [ 0.288095] acpiphp: Slot [20] registered [ 0.290172] acpiphp: Slot [21] registered [ 0.292188] acpiphp: Slot [22] registered [ 0.294103] acpiphp: Slot [23] registered [ 0.296130] acpiphp: Slot [24] registered [ 0.298113] acpiphp: Slot [25] registered [ 0.301122] acpiphp: Slot [26] registered [ 0.304442] acpiphp: Slot [27] registered [ 0.308264] acpiphp: Slot [28] registered [ 0.313147] acpiphp: Slot [29] registered [ 0.315145] acpiphp: Slot [30] registered [ 0.317117] acpiphp: Slot [31] registered [ 0.320082] PCI host bridge to bus 0000:00 [ 0.322023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.338029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.341026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343266] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.350294] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359801] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364640] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373546] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376802] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382568] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.418010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427204] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.444108] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.461017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.473016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.507017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522050] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.554014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.581016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.604330] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.617092] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.628017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.660016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.684364] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.695024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.703024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.724025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.731000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.739023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.751024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.768025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779673] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.790016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.803018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.822020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.836334] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.840502] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.844428] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.847500] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.850361] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.856062] iommu: Default domain type: Passthrough [ 0.857340] SCSI subsystem initialized [ 0.859210] ACPI: bus type USB registered [ 0.861128] usbcore: registered new interface driver usbfs [ 0.863113] usbcore: registered new interface driver hub [ 0.865141] usbcore: registered new device driver usb [ 0.868223] pps_core: LinuxPPS API ver. 1 registered [ 0.870014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.874086] PTP clock support registered [ 0.876199] EDAC MC: Ver: 3.0.0 [ 0.877000] PCI: Using ACPI for IRQ routing [ 0.878230] NetLabel: Initializing [ 0.880019] NetLabel: domain hash size = 128 [ 0.882015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.884178] NetLabel: unlabeled traffic allowed by default [ 0.887458] vgaarb: loaded [ 0.889848] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.892017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.902263] clocksource: Switched to clocksource kvm-clock [ 1.031347] VFS: Disk quotas dquot_6.6.0 [ 1.034290] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.037638] *** VALIDATE ramfs *** [ 1.039424] *** VALIDATE hugetlbfs *** [ 1.041580] pnp: PnP ACPI init [ 1.044750] pnp: PnP ACPI: found 6 devices [ 1.064163] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.068406] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.071097] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.073944] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.077068] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.080303] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.086177] NET: Registered protocol family 2 [ 1.091152] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.102402] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.108988] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.117960] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.126668] TCP: Hash tables configured (established 65536 bind 65536) [ 1.134524] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.137955] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.141611] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.145181] NET: Registered protocol family 1 [ 1.150698] RPC: Registered named UNIX socket transport module. [ 1.157381] RPC: Registered udp transport module. [ 1.161824] RPC: Registered tcp transport module. [ 1.167026] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.173606] NET: Registered protocol family 44 [ 1.177563] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.184507] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.190825] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.196035] PCI: CLS 0 bytes, default 64 [ 1.199463] Unpacking initramfs... [ 3.938540] debug: unmapping init [mem 0xffff9e103cc54000-0xffff9e103ffbffff] [ 3.956918] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.963249] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.970493] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.539870] Initialise system trusted keyrings [ 5.542268] Key type blacklist registered [ 5.544742] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.558572] zbud: loaded [ 5.562902] *** VALIDATE nfs *** [ 5.564351] *** VALIDATE nfs4 *** [ 5.566380] pstore: using deflate compression [ 5.571272] Platform Keyring initialized [ 5.705946] NET: Registered protocol family 38 [ 5.708483] Key type asymmetric registered [ 5.710330] Asymmetric key parser 'x509' registered [ 5.713593] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.717672] io scheduler mq-deadline registered [ 5.719765] io scheduler kyber registered [ 5.725369] io scheduler bfq registered [ 5.727627] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.731625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.734680] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.738565] ACPI: Power Button [PWRF] [ 5.746763] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.756624] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.774832] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.786312] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.807675] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.836810] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.869464] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.875405] Non-volatile memory driver v1.3 [ 5.877534] Linux agpgart interface v0.103 [ 5.909768] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 5.914510] vda: detected capacity change from 0 to 74694656 [ 5.929929] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.933747] vdb: detected capacity change from 0 to 1073741824 [ 5.958493] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.962784] vdc: detected capacity change from 0 to 2621440000 [ 5.981285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.984880] vdd: detected capacity change from 0 to 2621440000 [ 6.002882] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.006786] vde: detected capacity change from 0 to 4294967296 [ 6.024282] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.027915] vdf: detected capacity change from 0 to 4294967296 [ 6.036710] libphy: Fixed MDIO Bus: probed [ 6.044603] usbcore: registered new interface driver usbserial_generic [ 6.047659] usbserial: USB Serial support registered for generic [ 6.052706] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.061724] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.066721] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.071256] mousedev: PS/2 mouse device common for all mice [ 6.078318] rtc_cmos 00:05: RTC can wake from S4 [ 6.083111] rtc_cmos 00:05: registered as rtc0 [ 6.084655] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.087397] intel_pstate: CPU model not supported [ 6.090745] hid: raw HID events driver (C) Jiri Kosina [ 6.092449] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.093289] usbcore: registered new interface driver usbhid [ 6.093294] usbhid: USB HID core driver [ 6.093434] drop_monitor: Initializing network drop monitor service [ 6.093585] Initializing XFRM netlink socket [ 6.093958] NET: Registered protocol family 10 [ 6.095320] Segment Routing with IPv6 [ 6.095365] NET: Registered protocol family 17 [ 6.098840] mpls_gso: MPLS GSO support [ 6.103372] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.115433] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.132911] RAS: Correctable Errors collector initialized. [ 6.135449] AVX version of gcm_enc/dec engaged. [ 6.137471] AES CTR mode by8 optimization enabled [ 6.284907] sched_clock: Marking stable (6284881897, 0)->(7505353584, -1220471687) [ 6.290626] registered taskstats version 1 [ 6.292643] Loading compiled-in X.509 certificates [ 6.297437] zswap: loaded using pool lzo/zbud [ 6.329479] Key type big_key registered [ 6.345472] Key type encrypted registered [ 6.347421] ima: No TPM chip found, activating TPM-bypass! [ 6.350129] ima: Allocated hash algorithm: sha1 [ 6.352207] ima: No architecture policies found [ 6.354319] evm: Initialising EVM extended attributes: [ 6.356878] evm: security.selinux [ 6.358571] evm: security.ima [ 6.360140] evm: security.capability [ 6.361786] evm: HMAC attrs: 0x1 [ 6.366614] rtc_cmos 00:05: setting system clock to 2026-08-08 05:10:39 UTC (1786165839) [ 6.375930] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 6.382871] debug: unmapping init [mem 0xffffffff84182000-0xffffffff84458fff] [ 6.395139] Write protecting the kernel read-only data: 28672k [ 6.398473] debug: unmapping init [mem 0xffffffff82803000-0xffffffff829fffff] [ 6.403116] debug: unmapping init [mem 0xffffffff83114000-0xffffffff831fffff] [ 6.497110] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.507367] systemd[1]: Detected virtualization kvm. [ 6.509847] systemd[1]: Detected architecture x86-64. [ 6.511897] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.540845] systemd[1]: No hostname configured. [ 6.542968] systemd[1]: Set hostname to . [ 6.545046] random: systemd: uninitialized urandom read (16 bytes read) [ 6.548051] systemd[1]: Initializing machine ID from random generator. [ 6.679810] random: ln: uninitialized urandom read (6 bytes read) [ 7.154659] random: systemd: uninitialized urandom read (16 bytes read) [ 7.158714] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.165499] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.183558] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.790505] device-mapper: uevent: version 1.0.3 [ 8.796652] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.316426] virtio_net virtio0 ens2: renamed from eth0 [ 11.480527] random: fast init done [ 12.241597] scsi host0: ata_piix [ 12.461488] scsi host1: ata_piix [ 12.471486] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.481563] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.626667] random: crng init done [ 17.631074] random: 7 urandom warning(s) missed due to ratelimiting [ 20.008395] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.334825] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.156494] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.892452] SELinux: Disabled at runtime. [ 25.974510] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.985125] systemd[1]: Detected virtualization kvm. [ 25.986961] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.230703] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.244701] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.266418] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.276707] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.296770] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.338975] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.393093] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ 28.311880] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 29.938575] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.552941] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.659039] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.924354] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.109730] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.346711] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 37.959913] NFS: Registering the id_resolver key type [ 37.965383] Key type id_resolver registered [ 37.968493] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 55.017240] hrtimer: interrupt took 2642363 ns [ 90.383476] libcfs: loading out-of-tree module taints kernel. [ 90.434315] Key type ._llcrypt registered [ 90.436387] Key type .llcrypt registered [ 90.516829] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 107.170475] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 109.965553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 109.975546] alg: No test for adler32 (adler32-zlib) [ 111.380364] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 112.107350] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 113.871251] Key type lgssc registered [ 115.689428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 130.867340] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 168.743401] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 182.801434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.836964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.235841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.266287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.328410] Lustre: lustre-MDT0000: new disk, initializing [ 184.378750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.390473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.417956] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.474134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.664376] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.692866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.696543] Lustre: Skipped 1 previous similar message [ 202.887809] Lustre: lustre-MDT0001: new disk, initializing [ 203.042701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.076957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.087249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.478684] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.408270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.343247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.563982] Lustre: lustre-OST0000: new disk, initializing [ 222.567538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.572094] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.642881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.125760] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.413534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.420699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.472757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.101281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.213541] Lustre: lustre-OST0001: new disk, initializing [ 244.216537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.221016] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.280799] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.536989] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.955198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.981620] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 252.173080] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 267.387831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 273.575225] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.150733] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 285.251577] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 289.176363] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 291.924696] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 294.012521] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 295.447441] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 8 01:15:25 EDT 2026 [ 311.785383] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 313.764560] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.186470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.199321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.205375] Lustre: Skipped 1 previous similar message [ 329.219517] Lustre: Skipped 3 previous similar messages [ 334.340365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.348362] Lustre: Skipped 3 previous similar messages [ 335.417603] Lustre: server umount lustre-MDT0000 complete [ 339.634816] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166172 with bad export cookie 13267582144842378251 [ 339.642408] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.644352] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 344.543604] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.544488] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.544746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.551708] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 344.566602] Lustre: Skipped 3 previous similar messages [ 345.964756] Lustre: server umount lustre-MDT0001 complete [ 350.392536] Lustre: server umount lustre-OST0000 complete [ 354.401492] Lustre: server umount lustre-OST0001 complete [ 361.112865] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 369.859488] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 405.098690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.273179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.297748] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.350402] Lustre: lustre-MDT0000: new disk, initializing [ 405.399780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.412167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.001830] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 421.589771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.668236] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.710740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.714202] Lustre: Skipped 1 previous similar message [ 421.765200] Lustre: lustre-MDT0001: new disk, initializing [ 421.825420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.860240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.874390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.178980] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 435.753894] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.726557] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.231982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.392805] Lustre: lustre-OST0000: new disk, initializing [ 447.396571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.401582] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.504927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.640425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.648115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.782244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 452.803802] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 462.232253] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 467.197752] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 467.524681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 471.912500] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 472.070769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.601073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.612160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.632255] Lustre: Skipped 1 previous similar message [ 474.646828] Lustre: Skipped 2 previous similar messages [ 479.876176] Lustre: server umount lustre-OST0000 complete [ 483.297870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.305180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.316547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.584560] Lustre: server umount lustre-MDT0000 complete [ 489.952550] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.960792] LustreError: 15093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 493.684649] LustreError: 17049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166326 with bad export cookie 13267582144842379784 [ 493.697396] LustreError: 17049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 493.700924] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.952357] Lustre: server umount lustre-MDT0001 complete [ 497.849921] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:18:48 (1786166328) [ 500.720844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 502.326762] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:18:52 (1786166332) [ 509.317362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.862055] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.921211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.088575] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 515.082467] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.162029] LustreError: 19459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.138825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 527.860588] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 535.622340] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 541.392744] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.943642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 555.492187] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 562.035544] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 569.826555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.831138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.837465] Lustre: Skipped 1 previous similar message [ 569.841353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.846854] Lustre: Skipped 2 previous similar messages [ 574.709381] Lustre: server umount lustre-OST0000 complete [ 579.553463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.558897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.567396] Lustre: Skipped 1 previous similar message [ 584.632530] Lustre: server umount lustre-MDT0000 complete [ 587.232863] LustreError: 20940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.244510] LustreError: 20940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 588.700081] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166421 with bad export cookie 13267582144842380666 [ 588.706318] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.707278] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 588.930519] Lustre: server umount lustre-MDT0001 complete [ 599.047305] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 601.434587] Key type lgssc unregistered [ 601.641955] LNet: 22994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.649712] LNetError: 22994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.662116] LNet: Removed LNI 192.168.203.157@tcp [ 602.456265] Key type .llcrypt unregistered [ 602.459623] Key type ._llcrypt unregistered [ 623.711576] Key type ._llcrypt registered [ 623.713327] Key type .llcrypt registered [ 623.821275] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 625.777174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 625.800652] alg: No test for adler32 (adler32-zlib) [ 626.813972] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 627.092614] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 628.775683] Key type lgssc registered [ 629.674399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 638.264264] loop: module loaded [ 638.353466] loop0: detected capacity change from 0 to 204800000 [ 638.780934] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.807681] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.095337] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.526744] loop0: detected capacity change from 0 to 204800000 [ 644.712234] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.721096] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.736591] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 649.471917] loop0: detected capacity change from 0 to 204800000 [ 653.992909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.020343] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.189316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 655.241522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 655.298609] Lustre: lustre-MDT0000: new disk, initializing [ 655.537769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.562211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 659.745602] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 668.099271] loop1: detected capacity change from 0 to 204800000 [ 673.288781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 673.681261] Lustre: lustre-OST07c6: new disk, initializing [ 673.689719] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 673.696110] Lustre: Skipped 1 previous similar message [ 673.700851] Lustre: 26173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 673.792589] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 680.244819] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 680.609403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 680.619257] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 680.662230] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 684.459047] Lustre: 25160:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 684.468591] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 684.475663] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 684.483275] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 684.491227] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 684.498300] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 696.300417] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 696.309086] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.325464] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.341497] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 696.355177] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 696.368380] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.038682] Lustre: 26182:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 701.045184] Lustre: 26182:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 701.049336] Lustre: 26182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 701.053335] Lustre: 26182:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.059359] Lustre: 26182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.074719] Lustre: 26182:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.079404] Lustre: 26182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 701.084696] Lustre: 26182:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.089186] Lustre: 26182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.093879] Lustre: 26182:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.098507] Lustre: 26182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.102912] Lustre: 26182:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.424755] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 704.432897] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 704.443619] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 704.458496] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.470176] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.482913] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.495880] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 704.512601] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.529468] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 704.542880] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.558407] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.573130] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.442703] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 709.448170] Lustre: 25159:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 709.453365] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 709.459432] Lustre: 25159:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.511057] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 709.518640] Lustre: 25159:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.524271] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 709.530128] Lustre: 25159:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.534859] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 709.539274] Lustre: 25159:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 709.543909] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 709.548702] Lustre: 25159:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.623570] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 720.639745] Lustre: 25158:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 720.657198] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.668393] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.678291] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.690523] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.703542] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 720.717270] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.727706] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.743659] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.750048] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.761337] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 768.481972] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 768.486482] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 768.493618] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 772.584041] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 774.525445] Lustre: server umount lustre-OST07c6 complete [ 784.562504] Lustre: server umount lustre-MDT0000 complete [ 801.760250] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 804.653603] Key type lgssc unregistered [ 804.946704] LNet: 28731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.972454] LNetError: 28731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.998706] LNet: Removed LNI 192.168.203.157@tcp [ 805.881662] Key type .llcrypt unregistered [ 805.883832] Key type ._llcrypt unregistered [ 825.493878] Key type ._llcrypt registered [ 825.496618] Key type .llcrypt registered [ 825.607141] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 826.989272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.091082] alg: No test for adler32 (adler32-zlib) [ 828.049994] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 828.523341] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 830.143183] Key type lgssc registered [ 831.955298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.990138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 839.001345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.655259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 844.394912] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 849.962792] LustreError: 30006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.985778] LustreError: 30006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 854.025754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.290094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 858.521807] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 865.280330] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.886925] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 879.241184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.612470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.742722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 886.171914] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 894.116575] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.401473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.419164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 914.420421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 914.421215] Lustre: Skipped 1 previous similar message [ 914.432257] Lustre: Skipped 1 previous similar message [ 918.650653] Lustre: server umount lustre-OST0000 complete [ 924.640643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.641764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.653470] Lustre: Skipped 1 previous similar message [ 924.669085] Lustre: Skipped 1 previous similar message [ 928.078356] Lustre: server umount lustre-MDT0000 complete [ 929.764043] LustreError: 32565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.785214] LustreError: 32565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 931.981471] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166765 with bad export cookie 12180737363977399334 [ 931.993896] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.999284] LustreError: 29987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 932.302329] Lustre: server umount lustre-MDT0001 complete [ 943.817666] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 946.410932] Key type lgssc unregistered [ 946.796664] LNet: 33750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.814059] LNetError: 33750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.835637] LNet: Removed LNI 192.168.203.157@tcp [ 947.623516] Key type .llcrypt unregistered [ 947.626416] Key type ._llcrypt unregistered [ 965.469476] Key type ._llcrypt registered [ 965.477061] Key type .llcrypt registered [ 965.718801] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 966.842644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.944284] alg: No test for adler32 (adler32-zlib) [ 967.990987] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 968.184656] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 969.903628] Key type lgssc registered [ 970.851313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.434646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.445735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.756830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.529866] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 989.170742] LustreError: 35020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.183317] LustreError: 35020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 992.349829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.569535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 997.214110] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1004.599750] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.785766] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1017.843158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.070559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.242311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1024.614295] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1032.364984] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1048.041471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1048.046620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.049322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1048.055802] Lustre: Skipped 1 previous similar message [ 1048.063763] Lustre: Skipped 1 previous similar message [ 1051.494830] Lustre: server umount lustre-OST0000 complete [ 1058.275177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.277035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.306082] Lustre: Skipped 1 previous similar message [ 1058.376381] Lustre: Skipped 1 previous similar message [ 1061.812230] Lustre: server umount lustre-MDT0000 complete [ 1063.399333] LustreError: 35749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.440367] LustreError: 35749:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1065.461722] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786166898 with bad export cookie 3524056257029761885 [ 1065.472614] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1065.473227] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1065.806451] Lustre: server umount lustre-MDT0001 complete [ 1078.066536] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1080.634550] Key type lgssc unregistered [ 1081.324422] LNet: 38578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.342158] LNetError: 38578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.357916] LNet: Removed LNI 192.168.203.157@tcp [ 1082.254730] Key type .llcrypt unregistered [ 1082.257171] Key type ._llcrypt unregistered [ 1109.608697] Key type ._llcrypt registered [ 1109.610412] Key type .llcrypt registered [ 1109.705748] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1110.691835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.888708] alg: No test for adler32 (adler32-zlib) [ 1112.018208] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1112.335367] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1114.023790] Key type lgssc registered [ 1115.218751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1122.447430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1122.465773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.718696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.237717] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1133.092595] LustreError: 39991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1133.113807] LustreError: 39991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1136.824595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.366224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1142.766442] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1152.267808] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1157.240617] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1165.303725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1165.590061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.824211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1172.537757] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1179.714216] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.296170] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1193.440516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1193.444456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.958371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.967866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.972143] Lustre: Skipped 1 previous similar message [ 1193.981935] Lustre: Skipped 1 previous similar message [ 1196.485202] Lustre: server umount lustre-MDT0000 complete [ 1198.423733] LustreError: 39988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.829984] LustreError: 39976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167032 with bad export cookie 4431876081708078897 [ 1199.838593] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1199.845404] LustreError: 39976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1203.516603] LustreError: 41836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.528764] LustreError: 41836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1203.540404] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 1203.559132] Lustre: Skipped 1 previous similar message [ 1204.194796] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.317196] Lustre: server umount lustre-MDT0001 complete [ 1228.504260] Lustre: server umount lustre-OST0000 complete [ 1246.091903] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1248.125867] Key type lgssc unregistered [ 1248.385118] LNet: 43858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.409037] LNetError: 43858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.464656] LNet: Removed LNI 192.168.203.157@tcp [ 1250.307402] Key type .llcrypt unregistered [ 1250.309544] Key type ._llcrypt unregistered [ 1278.828691] Key type ._llcrypt registered [ 1278.830832] Key type .llcrypt registered [ 1278.930915] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1280.815123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.925331] alg: No test for adler32 (adler32-zlib) [ 1282.033675] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1282.219842] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1283.863323] Key type lgssc registered [ 1284.785262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.051384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.069494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.371485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.301368] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1303.523606] LustreError: 45275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.545294] LustreError: 45275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1306.021508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1306.427437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1310.364246] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1317.796147] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1323.410624] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1331.270881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1331.568993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1338.743805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1339.189820] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1346.944481] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1359.330397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1359.340388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.406425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1360.206953] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1361.377661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1361.385158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1361.387079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1361.400149] Lustre: Skipped 1 previous similar message [ 1364.110697] Lustre: server umount lustre-OST0000 complete [ 1383.395420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1383.421581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.444767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.980625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.851860] Lustre: server umount lustre-MDT0000 complete [ 1392.154283] LustreError: 47765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.165610] LustreError: 47765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1393.341073] LustreError: 45255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167226 with bad export cookie 15683394259943168805 [ 1393.349155] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.354984] LustreError: 45255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1399.794825] Lustre: server umount lustre-MDT0001 complete [ 1412.147528] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1415.028742] Key type lgssc unregistered [ 1415.264018] LNet: 48991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.272862] LNetError: 48991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.297372] LNet: Removed LNI 192.168.203.157@tcp [ 1416.051388] Key type .llcrypt unregistered [ 1416.053494] Key type ._llcrypt unregistered [ 1441.522750] Key type ._llcrypt registered [ 1441.524950] Key type .llcrypt registered [ 1441.621546] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1443.565955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.641646] alg: No test for adler32 (adler32-zlib) [ 1444.707655] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1444.966311] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1446.639317] Key type lgssc registered [ 1447.662903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1455.954211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1455.970177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.265971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.231148] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1467.882421] LustreError: 50412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1467.936743] LustreError: 50412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1470.004131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.305403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1474.199557] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1482.169137] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1491.021250] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1501.171429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.611925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.687629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1507.025581] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1515.304395] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1521.909899] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1525.494207] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1537.866565] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-08 1h36m07s [ 1539.934034] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1544.065455] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1545.094040] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1545.743807] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.764882] Lustre: Skipped 2 previous similar messages [ 1546.788231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1546.803128] Lustre: Skipped 5 previous similar messages [ 1549.875597] Lustre: server umount lustre-MDT0000 complete [ 1550.141737] LustreError: 51650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.929378] LustreError: 51844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.960307] LustreError: 51844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1554.152414] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1556.281662] LustreError: 50406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1560.789572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.894356] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.255198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.366869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1562.318771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1566.123720] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1569.670469] Lustre: DEBUG MARKER: Wait for df (36690) ... [ 1571.708668] Lustre: DEBUG MARKER: done [ 1576.420587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1576.421426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.426781] LustreError: Skipped 1 previous similar message [ 1576.428406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1576.441224] Lustre: Skipped 1 previous similar message [ 1581.511987] Lustre: server umount lustre-OST0000 complete [ 1588.205932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1588.207760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.220047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.223076] Lustre: Skipped 1 previous similar message [ 1588.230982] Lustre: Skipped 4 previous similar messages [ 1591.401671] Lustre: server umount lustre-MDT0000 complete [ 1595.426268] LustreError: 50412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.458392] LustreError: 50412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1595.602824] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167428 with bad export cookie 16361365483558847118 [ 1595.606974] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1595.608919] LustreError: 50392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1596.114382] Lustre: server umount lustre-MDT0001 complete [ 1606.796652] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1609.148154] Key type lgssc unregistered [ 1609.529115] LNet: 55339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.533812] LNetError: 55339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.598249] LNet: Removed LNI 192.168.203.157@tcp [ 1611.245383] Key type .llcrypt unregistered [ 1611.247381] Key type ._llcrypt unregistered [ 1655.507497] Key type ._llcrypt registered [ 1655.510579] Key type .llcrypt registered [ 1655.593608] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1658.489344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1658.528164] alg: No test for adler32 (adler32-zlib) [ 1659.514526] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1659.758133] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1661.393135] Key type lgssc registered [ 1663.023034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1671.421488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1671.431616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.677898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1672.691712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1672.817727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.858558] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1686.129232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.194802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1686.220645] Lustre: 57552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1686.399662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.278966] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1698.434702] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1703.835848] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1711.090332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1711.228490] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1711.464609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.593294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1719.603126] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1728.220585] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.112850] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-08 1h39m25s [ 1737.642643] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1744.655426] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1747.514652] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1747.623099] LustreError: 57557:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1767.820803] Lustre: lustre-MDT0000: Client 02bde19a-205d-410a-a3a1-b5147704fcd0 (at 192.168.203.57@tcp) reconnecting [ 1787.685589] LustreError: 57557:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1790.024836] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1792.992903] Lustre: DEBUG MARKER: done [ 1800.675847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.677173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1800.678175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.691436] Lustre: Skipped 1 previous similar message [ 1800.695826] LustreError: Skipped 1 previous similar message [ 1800.775912] Lustre: server umount lustre-OST0000 complete [ 1804.257222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1804.265971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.289399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.294330] Lustre: Skipped 1 previous similar message [ 1805.351267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.362717] Lustre: Skipped 2 previous similar messages [ 1810.403393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1810.426353] Lustre: Skipped 1 previous similar message [ 1810.636950] Lustre: server umount lustre-MDT0000 complete [ 1814.937799] LustreError: 57543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167648 with bad export cookie 11106276820099423092 [ 1814.982565] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.071315] LustreError: 57543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1815.458979] Lustre: server umount lustre-MDT0001 complete [ 1827.144795] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1829.808754] Key type lgssc unregistered [ 1830.216995] LNet: 61888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.222531] LNetError: 61888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.270677] LNet: Removed LNI 192.168.203.157@tcp [ 1832.066509] Key type .llcrypt unregistered [ 1832.068452] Key type ._llcrypt unregistered [ 1877.992671] Key type ._llcrypt registered [ 1877.994180] Key type .llcrypt registered [ 1878.101302] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1879.118524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.239075] alg: No test for adler32 (adler32-zlib) [ 1880.443346] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1881.035770] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1882.711316] Key type lgssc registered [ 1883.542693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.101374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.150207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.377848] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1887.390286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.486685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.442112] Lustre: Failing over lustre-MDT0000 [ 1888.617806] Lustre: server umount lustre-MDT0000 complete [ 1896.387825] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:42:06 (1786167726) [ 1902.291562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.550475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.234807] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1914.761223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.856124] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1914.874490] Lustre: 64689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1915.094747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1919.734228] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1926.881555] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1932.311489] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1940.561395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1940.684562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1940.846567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.660287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1947.391580] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1953.748657] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1960.204414] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1966.611086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1966.615139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.626464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.568972] Lustre: server umount lustre-MDT0000 complete [ 1970.889031] LustreError: 65425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167804 with bad export cookie 10982315919195850994 [ 1970.892302] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1970.901874] LustreError: 65425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1971.142898] Lustre: server umount lustre-MDT0001 complete [ 1974.941693] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1979.060080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1986.058440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.575488] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167805/real 1786167805] req@00000000a221e5aa x1872932596912000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1988.617322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.635743] Lustre: Skipped 2 previous similar messages [ 1991.715732] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167808/real 1786167808] req@0000000045cc79eb x1872932596912128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1993.695161] Lustre: 63610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167810/real 1786167810] req@0000000064e1c871 x1872932596912256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.872415] LustreError: 63609:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000410ac2c2 x1872932596913152/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1996.216634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.300890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1997.215183] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167814/real 1786167814] req@000000008cc1dee2 x1872932596912512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.998402] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2002.336539] Lustre: 63612:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167819/real 1786167819] req@00000000ac017fd3 x1872932596912640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.435931] LustreError: 68459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.460342] LustreError: 68459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2007.526323] LustreError: 68461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.537190] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2007.539429] LustreError: 68461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2009.092454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.322519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.601249] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2014.704985] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2021.484462] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2027.457516] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.866934] Lustre: server umount lustre-MDT0000 complete [ 2035.168053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.172899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.184978] LustreError: 66540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.192383] Lustre: Skipped 1 previous similar message [ 2035.214549] LustreError: 66540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2036.863687] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167869 with bad export cookie 10982315919195853276 [ 2036.870409] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2036.876658] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2037.098376] Lustre: server umount lustre-MDT0001 complete [ 2043.564926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2053.075421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.420838] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786167873/real 1786167873] req@00000000a87b4fb9 x1872932596938112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786167889 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2056.437925] Lustre: 63611:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2056.444837] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.452691] Lustre: Skipped 1 previous similar message [ 2062.933670] LustreError: 69194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.013610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.120265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2068.361944] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2071.529094] LustreError: 71140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.549385] LustreError: 71140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2075.654808] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.970948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.287815] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2082.351598] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2089.404083] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2094.294257] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2102.470105] Lustre: server umount lustre-MDT0000 complete [ 2102.751988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2102.759971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.770575] Lustre: Skipped 1 previous similar message [ 2102.772368] LustreError: 72626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.792737] LustreError: 72626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2105.783068] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786167938 with bad export cookie 10982315919195854655 [ 2105.799585] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.812534] LustreError: 64680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2106.285345] Lustre: server umount lustre-MDT0001 complete [ 2112.643761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2115.688249] Lustre: DEBUG MARKER: files compared the same [ 2128.311360] Lustre: server umount lustre-OST0000 complete [ 2146.391687] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2148.528546] Key type lgssc unregistered [ 2148.762123] LNet: 74419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.770240] LNetError: 74419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.788150] LNet: Removed LNI 192.168.203.157@tcp [ 2149.706834] Key type .llcrypt unregistered [ 2149.709051] Key type ._llcrypt unregistered [ 2172.993463] Key type ._llcrypt registered [ 2173.000167] Key type .llcrypt registered [ 2173.101981] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2174.951687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.983204] alg: No test for adler32 (adler32-zlib) [ 2176.177430] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2176.362992] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2177.999346] Key type lgssc registered [ 2179.024860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2185.642523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2185.665370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.030729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.225368] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug +malloc all [ 2197.476329] LustreError: 75837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2197.490237] LustreError: 75837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2199.878457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.116594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.208128] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug +malloc all [ 2212.271897] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2218.325217] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2225.441709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2225.668649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.527319] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug +malloc all [ 2232.833754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2238.653509] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2248.163446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2248.166812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.201791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.210120] Lustre: Skipped 1 previous similar message [ 2253.285130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.290074] Lustre: Skipped 2 previous similar messages [ 2253.455757] Lustre: server umount lustre-OST0000 complete [ 2258.407970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.409371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.417791] Lustre: Skipped 1 previous similar message [ 2258.424518] Lustre: Skipped 1 previous similar message [ 2263.523876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.536156] Lustre: Skipped 1 previous similar message [ 2263.678648] Lustre: server umount lustre-MDT0000 complete [ 2267.139496] LustreError: 75817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168100 with bad export cookie 2256821686625110673 [ 2267.142438] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.145498] LustreError: 75817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2267.458606] Lustre: server umount lustre-MDT0001 complete [ 2278.591613] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2280.948900] Key type lgssc unregistered [ 2281.321388] LNet: 79374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.336819] LNetError: 79374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.357505] LNet: Removed LNI 192.168.203.157@tcp [ 2282.003667] Key type .llcrypt unregistered [ 2282.006249] Key type ._llcrypt unregistered [ 2307.369261] Key type ._llcrypt registered [ 2307.371107] Key type .llcrypt registered [ 2307.499314] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2309.578652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.593787] alg: No test for adler32 (adler32-zlib) [ 2310.546942] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2310.701671] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2312.359085] Key type lgssc registered [ 2313.409153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2320.093598] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2320.104610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.583401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2340.703343] LustreError: 80766:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2367.455259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2373.859477] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2381.739191] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2389.730490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.848116] Lustre: *** cfs_fail_loc=706, val=0*** [ 2390.194393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.335527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2395.618826] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.632545] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2396.491923] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2400.739663] LustreError: 80791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.788471] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.798512] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2406.002207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2406.289901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2410.886671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2419.308389] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2425.717549] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2431.456158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.465609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.472706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.975497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.985886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.032386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2433.911556] Lustre: server umount lustre-OST0000 complete [ 2437.091100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.094836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2437.095034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.099370] Lustre: Skipped 1 previous similar message [ 2442.209187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.214700] Lustre: Skipped 2 previous similar messages [ 2443.146722] Lustre: server umount lustre-MDT0000 complete [ 2447.211315] LustreError: 80774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168280 with bad export cookie 8931987358145247706 [ 2447.236772] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.239520] LustreError: 80774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2447.488975] Lustre: server umount lustre-MDT0001 complete [ 2459.537671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2462.295560] Key type lgssc unregistered [ 2462.540995] LNet: 84384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.556751] LNetError: 84384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.580779] LNet: Removed LNI 192.168.203.157@tcp [ 2463.447841] Key type .llcrypt unregistered [ 2463.449937] Key type ._llcrypt unregistered [ 2490.242851] Key type ._llcrypt registered [ 2490.250383] Key type .llcrypt registered [ 2490.360771] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2491.641126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.671096] alg: No test for adler32 (adler32-zlib) [ 2492.738975] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2493.004350] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2494.655127] Key type lgssc registered [ 2495.839383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.375370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.387491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.370744] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2515.184388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.605562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.318472] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2521.072979] LustreError: 86488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.101971] LustreError: 86488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2525.156445] LustreError: 86492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.182778] LustreError: 86492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2527.864091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.079136] LustreError: 86488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.102212] LustreError: 86488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2528.121084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.368286] LustreError: 86491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.392300] LustreError: 86491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2534.412323] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2538.466037] LustreError: 86486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.493664] LustreError: 86486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2543.300961] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.326138] Lustre: MGS: Client 5ebddc28-3b06-4a8b-8753-cdf1c7e9a92f (at 0@lo) reconnecting [ 2543.357971] LustreError: 86502:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2543.531216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.604341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2548.310676] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2564.067411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2564.068487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2564.089931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2564.102192] Lustre: Skipped 1 previous similar message [ 2567.807775] Lustre: server umount lustre-OST0000 complete [ 2574.309184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2574.310598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.311484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.311490] Lustre: Skipped 1 previous similar message [ 2574.314620] Lustre: Skipped 1 previous similar message [ 2577.884068] Lustre: server umount lustre-MDT0000 complete [ 2579.429029] LustreError: 86487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.455750] LustreError: 86487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2581.934345] Lustre: server umount lustre-MDT0001 complete [ 2585.750603] Lustre: server umount MGS complete [ 2599.287844] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:53:49 (1786168429) [ 2627.551690] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 2637.551347] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2670.923669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.559890] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2683.961236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.144885] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2684.179956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2684.233345] Lustre: lustre-MDT0001: new disk, initializing [ 2684.323793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.481727] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2684.518172] Lustre: 93656:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2684.546909] LustreError: 93656:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2688.472385] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2697.944459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.179459] Lustre: lustre-OST0000: new disk, initializing [ 2698.183477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2698.191407] Lustre: 94422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2698.266499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.555595] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2709.091223] Lustre: 94939:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000002c0ded1d x1872933238644608/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786168558 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2712.727475] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2720.503191] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2720.515714] Lustre: MGS: Client ae44474a-cad7-4c23-b4a2-d341f9da39de (at 0@lo) reconnecting [ 2720.544493] LustreError: 93642:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2720.573228] Lustre: 92924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.604929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2720.665941] Lustre: lustre-MDT0000: new disk, initializing [ 2720.773531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2720.793957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2720.920624] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2721.888820] Lustre: 95490:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009a1fd4cf x1872933238651136/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786168571 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2726.599868] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2726.969927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2726.997371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2727.481292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2751.470651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.480710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.501921] Lustre: Skipped 1 previous similar message [ 2751.510666] Lustre: Skipped 1 previous similar message [ 2756.624221] Lustre: server umount lustre-OST0000 complete [ 2761.865523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.870436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.879916] Lustre: Skipped 1 previous similar message [ 2761.901396] Lustre: Skipped 2 previous similar messages [ 2766.449150] Lustre: server umount lustre-MDT0000 complete [ 2767.331990] LustreError: 95915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.367820] LustreError: 95915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2777.696514] Lustre: server umount lustre-MDT0001 complete [ 2781.932489] Lustre: server umount MGS complete [ 2794.904572] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:57:05 (1786168625) [ 2813.734811] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2816.310522] Key type lgssc unregistered [ 2816.518816] LNet: 97989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.523337] LNetError: 97989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.533845] LNet: Removed LNI 192.168.203.157@tcp [ 2817.508519] Key type .llcrypt unregistered [ 2817.513300] Key type ._llcrypt unregistered [ 2836.509712] Key type ._llcrypt registered [ 2836.513506] Key type .llcrypt registered [ 2836.610877] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2838.769774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.072502] alg: No test for adler32 (adler32-zlib) [ 2840.571965] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2841.071188] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2842.791541] Key type lgssc registered [ 2844.757694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2852.776769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2852.795147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.180824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.166959] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2864.613924] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.635389] LustreError: 99213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2867.536506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.833036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.411064] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2881.104962] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2887.427829] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2894.956968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2895.294357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.372351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2900.710593] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2908.449378] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2919.904414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2919.911309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.959851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.977071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2922.981542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2922.995915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2923.024972] Lustre: Skipped 1 previous similar message [ 2925.085086] Lustre: server umount lustre-OST0000 complete [ 2933.224393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.232608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.246407] Lustre: Skipped 1 previous similar message [ 2933.257686] Lustre: Skipped 1 previous similar message [ 2935.734627] Lustre: server umount lustre-MDT0000 complete [ 2938.339576] LustreError: 100453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.371746] LustreError: 100453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2939.571591] LustreError: 99194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168772 with bad export cookie 14269707376321163838 [ 2939.581951] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.590904] LustreError: 99194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2940.070792] Lustre: server umount lustre-MDT0001 complete [ 2951.515749] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2953.992430] Key type lgssc unregistered [ 2954.236544] LNet: 102752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2954.244511] LNetError: 102752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2955.302310] LNet: Removed LNI 192.168.203.157@tcp [ 2955.920886] Key type .llcrypt unregistered [ 2955.923286] Key type ._llcrypt unregistered [ 2973.658552] Key type ._llcrypt registered [ 2973.660405] Key type .llcrypt registered [ 2973.828873] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2975.157327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2975.501324] alg: No test for adler32 (adler32-zlib) [ 2977.304854] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2977.846747] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2979.727207] Key type lgssc registered [ 2981.778134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2989.242867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2989.246693] LustreError: 103837:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2989.251792] Lustre: Skipped 1 previous similar message [ 2989.251811] LustreError: 103824:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2989.269922] LustreError: 103837:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2989.274455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.281723] LustreError: 103837:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2989.299201] LustreError: 103837:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2990.797056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.339613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.528064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3003.833619] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3009.472127] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing lsmod [ 3013.151920] LustreError: 105026:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3013.412588] LustreError: 105044:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3013.443691] LustreError: 105026:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4722 [ 3013.453128] LustreError: 105026:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3013.486118] LustreError: 105026:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3013.497544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.119529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.269831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3020.265392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.273171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.287531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.318149] Lustre: Skipped 2 previous similar messages [ 3024.081037] Lustre: server umount lustre-MDT0000 complete [ 3024.420488] LustreError: 103869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.451938] LustreError: 103869:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3028.146240] LustreError: 103846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168861 with bad export cookie 14213115952520044873 [ 3028.147615] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.155967] LustreError: 103846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3028.615297] LustreError: 104306:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d497d377 x1872933746136192/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3028.835716] Lustre: server umount lustre-MDT0001 complete [ 3043.039720] Lustre: server umount lustre-OST0000 complete [ 3052.198939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.587204] LustreError: 106121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.658895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.886071] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3058.163864] LustreError: 106122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.243426] LustreError: 106121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.391906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.617514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3072.225480] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3080.903262] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3088.805969] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.336766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.689471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.715624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3102.219842] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3110.627946] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3118.566356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.582603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.600235] Lustre: Skipped 2 previous similar messages [ 3118.612678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.633734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3121.647176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3121.663309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.761329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3126.767961] Lustre: Skipped 2 previous similar messages [ 3131.873378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3131.880550] Lustre: Skipped 1 previous similar message [ 3132.383475] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3132.514686] Lustre: server umount lustre-OST0000 complete [ 3138.528152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3138.533133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.065967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.070062] Lustre: Skipped 2 previous similar messages [ 3143.956970] Lustre: server umount lustre-MDT0000 complete [ 3146.211892] LustreError: 106116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.241572] LustreError: 106116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3147.648025] LustreError: 106103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786168980 with bad export cookie 14213115952520045804 [ 3147.655140] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.657221] LustreError: 106103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3147.903235] Lustre: server umount lustre-MDT0001 complete [ 3160.818436] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3163.444178] Key type lgssc unregistered [ 3163.707813] LNet: 109660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3163.712871] LNetError: 109660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3163.738646] LNet: Removed LNI 192.168.203.157@tcp [ 3164.394787] Key type .llcrypt unregistered [ 3164.397217] Key type ._llcrypt unregistered [ 3189.131944] Key type ._llcrypt registered [ 3189.133940] Key type .llcrypt registered [ 3189.226616] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3190.404016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3190.647277] alg: No test for adler32 (adler32-zlib) [ 3192.012084] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3192.195638] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3193.849952] Key type lgssc registered [ 3194.718537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3202.466428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3202.481056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3203.804765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.584399] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3213.561104] LustreError: 111056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.571255] LustreError: 111056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3218.031640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.288165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3223.246713] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3232.122351] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3239.698726] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.679902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3246.888247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.929438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3253.364294] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3260.372171] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3266.648436] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3269.464031] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3273.699854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.701648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.729414] LustreError: Skipped 1 previous similar message [ 3273.770555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.791271] Lustre: Skipped 1 previous similar message [ 3275.104759] Lustre: server umount lustre-OST0000 complete [ 3278.818379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.830319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.837314] Lustre: Skipped 2 previous similar messages [ 3278.857267] Lustre: Skipped 1 previous similar message [ 3283.936729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.941770] Lustre: Skipped 1 previous similar message [ 3289.056865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3289.060069] Lustre: Skipped 1 previous similar message [ 3293.152428] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3293.154688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.162132] Lustre: Skipped 1 previous similar message [ 3293.264263] Lustre: server umount lustre-MDT0000 complete [ 3297.512321] LustreError: 111035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169130 with bad export cookie 7154201756074120072 [ 3297.524543] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.526843] LustreError: 111035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3297.870497] Lustre: server umount lustre-MDT0001 complete [ 3311.237253] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3314.304838] Key type lgssc unregistered [ 3314.599669] LNet: 114698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.606198] LNetError: 114698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.632544] LNet: Removed LNI 192.168.203.157@tcp [ 3315.796511] Key type .llcrypt unregistered [ 3315.799044] Key type ._llcrypt unregistered [ 3333.981260] Key type ._llcrypt registered [ 3333.985145] Key type .llcrypt registered [ 3334.061299] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3335.402329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3335.436726] alg: No test for adler32 (adler32-zlib) [ 3336.423875] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3336.661899] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3338.303193] Key type lgssc registered [ 3339.388233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3345.809607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3345.825749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.122431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.539792] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3357.669887] LustreError: 115923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.701349] LustreError: 115923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3361.294499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.528320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.223624] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3374.372626] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3381.102602] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.116809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.332898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.453317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3395.297177] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3402.892658] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3416.034439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3416.044284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.063537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.056324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.072546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.076568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.106908] Lustre: Skipped 1 previous similar message [ 3421.915544] Lustre: server umount lustre-OST0000 complete [ 3427.298154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.300817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.317800] Lustre: Skipped 1 previous similar message [ 3427.332228] Lustre: Skipped 1 previous similar message [ 3431.697641] Lustre: server umount lustre-MDT0000 complete [ 3432.417628] LustreError: 115924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.427136] LustreError: 115924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3435.288165] LustreError: 116657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169268 with bad export cookie 8739024982883481722 [ 3435.309864] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.329977] LustreError: 116657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3435.728850] Lustre: server umount lustre-MDT0001 complete [ 3447.643472] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3450.287917] Key type lgssc unregistered [ 3450.483866] LNet: 119480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3450.489439] LNetError: 119480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3450.499884] LNet: Removed LNI 192.168.203.157@tcp [ 3451.082191] Key type .llcrypt unregistered [ 3451.089178] Key type ._llcrypt unregistered [ 3476.344817] Key type ._llcrypt registered [ 3476.346499] Key type .llcrypt registered [ 3476.418763] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3477.716673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3477.806638] alg: No test for adler32 (adler32-zlib) [ 3478.908640] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3479.300122] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3481.047162] Key type lgssc registered [ 3482.055949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3488.771973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3488.784566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.284744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3494.263418] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3500.543777] LustreError: 120897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.568960] LustreError: 120897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3501.556456] LustreError: 120898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3504.413525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.763221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.911547] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3517.394406] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3523.203820] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3529.053400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3529.261589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.094593] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3536.446083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3543.505210] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3549.692745] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3553.462833] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3553.476172] Lustre: Skipped 2 previous similar messages [ 3557.744519] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3557.761435] Lustre: Skipped 2 previous similar messages [ 3559.992901] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3560.018958] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3565.542762] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.57@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3565.550270] Lustre: Skipped 2 previous similar messages [ 3571.269731] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.57@tcp 192.168.0.[2,10]@tcp [ 3576.073584] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.57/24@tcp in log lustre-MDT0000 [ 3576.079028] Lustre: Skipped 2 previous similar messages [ 3585.104919] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.57/24@tcp [ 3585.109236] Lustre: Skipped 1 previous similar message [ 3592.673087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3592.675839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.688716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3592.692329] Lustre: Skipped 1 previous similar message [ 3593.908511] Lustre: server umount lustre-OST0000 complete [ 3597.980905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.160928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.171487] Lustre: Skipped 2 previous similar messages [ 3603.424316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3603.434920] Lustre: Skipped 1 previous similar message [ 3604.175280] Lustre: server umount lustre-MDT0000 complete [ 3608.344529] LustreError: 120879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169441 with bad export cookie 6408388464988432791 [ 3608.350761] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.354564] LustreError: 120879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3608.616718] Lustre: server umount lustre-MDT0001 complete [ 3620.952746] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3623.359549] Key type lgssc unregistered [ 3623.601877] LNet: 125881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3623.606666] LNetError: 125881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3623.620140] LNet: Removed LNI 192.168.203.157@tcp [ 3624.337923] Key type .llcrypt unregistered [ 3624.339983] Key type ._llcrypt unregistered [ 3656.350657] Key type ._llcrypt registered [ 3656.390170] Key type .llcrypt registered [ 3656.579719] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3658.344837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3658.385204] alg: No test for adler32 (adler32-zlib) [ 3659.384482] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3659.710387] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3661.423168] Key type lgssc registered [ 3662.378238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3669.346813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3669.380728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.861371] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3670.954883] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.57/24@tcp [ 3671.011102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.332861] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3680.233505] LustreError: 127446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.253710] LustreError: 127446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3683.276303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.652114] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3683.718476] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.57/24@tcp [ 3683.805552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3687.650916] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3695.270121] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3701.119470] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3707.775276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.079430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3714.801266] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3715.450218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3721.786493] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3735.016747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3735.022964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.031355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.039121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.040816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.072284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.714241] Lustre: server umount lustre-OST0000 complete [ 3745.248625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3745.256794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.264795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3745.268592] Lustre: Skipped 1 previous similar message [ 3750.063625] Lustre: server umount lustre-MDT0000 complete [ 3751.393345] LustreError: 127440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3751.402919] LustreError: 127440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3754.084438] LustreError: 127426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169587 with bad export cookie 3685384365577688760 [ 3754.103721] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.105672] LustreError: 127426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3754.695687] Lustre: server umount lustre-MDT0001 complete [ 3765.921257] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3768.265673] Key type lgssc unregistered [ 3768.492795] LNet: 131036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.503543] LNetError: 131036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.522882] LNet: Removed LNI 192.168.203.157@tcp [ 3769.225918] Key type .llcrypt unregistered [ 3769.229576] Key type ._llcrypt unregistered [ 3800.180393] Key type ._llcrypt registered [ 3800.184640] Key type .llcrypt registered [ 3800.327607] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3815.528579] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3816.534804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3816.553221] alg: No test for adler32 (adler32-zlib) [ 3818.440725] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 3818.739140] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3820.583590] Key type lgssc registered [ 3821.464642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.089478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.142149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.334257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3855.363755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3855.416153] Lustre: lustre-MDT0000: new disk, initializing [ 3855.482798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.494881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3859.709795] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3870.407764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.525837] Lustre: 134431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3870.584633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3870.587976] Lustre: Skipped 1 previous similar message [ 3870.647628] Lustre: lustre-MDT0001: new disk, initializing [ 3870.834518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.890073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3870.895485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3875.533644] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3884.905315] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3890.981256] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3898.527057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3898.810940] Lustre: lustre-OST0000: new disk, initializing [ 3898.814243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3898.820944] Lustre: 136400:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3898.889574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.749902] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3907.318074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3907.329416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3907.426353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3915.703952] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3920.143990] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3920.576894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3925.541675] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3925.695470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3933.152346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.152996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.160282] LustreError: Skipped 1 previous similar message [ 3933.164508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.178379] Lustre: Skipped 1 previous similar message [ 3933.196632] Lustre: Skipped 1 previous similar message [ 3934.392352] Lustre: server umount lustre-OST0000 complete [ 3938.275344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.284149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3938.288536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.294730] Lustre: Skipped 1 previous similar message [ 3943.393640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.398087] Lustre: Skipped 2 previous similar messages [ 3944.572490] Lustre: server umount lustre-MDT0000 complete [ 3947.657193] LustreError: 134424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169780 with bad export cookie 17750201377166629109 [ 3947.665890] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.666245] LustreError: 134424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3947.961993] Lustre: server umount lustre-MDT0001 complete [ 3955.596461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3955.947256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.948457] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3961.323717] LustreError: 138620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.333942] LustreError: 138620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3966.457101] LustreError: 138619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.525799] LustreError: 138620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.504038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.769840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3974.099591] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3981.793407] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3987.926537] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3993.811774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3994.062734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.072602] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4008.651393] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4013.466935] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4013.682630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4017.968828] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4018.144916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4027.265570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4027.380956] Lustre: lustre-OST0001: new disk, initializing [ 4027.384774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4027.389796] Lustre: 141992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4027.459767] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4033.466790] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4033.609306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4033.621528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4033.750559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4042.412194] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4042.747785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4048.385463] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4048.591626] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4054.384368] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e52859e1000.ost_server_uuid 50 [ 4055.859934] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e52859e1000.ost_server_uuid in IDLE state after 0 sec [ 4067.519713] Lustre: server umount lustre-OST0001 complete [ 4069.857747] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.868948] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.889202] LustreError: 140444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.915928] LustreError: 140444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4073.980368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4073.984993] LustreError: Skipped 1 previous similar message [ 4073.987757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.994510] Lustre: Skipped 1 previous similar message [ 4073.998115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4074.977637] LustreError: 140444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.996803] LustreError: 140444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4077.491612] Lustre: server umount lustre-OST0000 complete [ 4081.635809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4081.639959] LustreError: Skipped 1 previous similar message [ 4081.642423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.648803] Lustre: Skipped 1 previous similar message [ 4081.651775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4081.655180] Lustre: Skipped 2 previous similar messages [ 4087.397855] Lustre: server umount lustre-MDT0000 complete [ 4090.366714] LustreError: 138620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.408137] LustreError: 138620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4091.420511] LustreError: 138600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786169924 with bad export cookie 17750201377166629998 [ 4091.429352] LustreError: 138600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4091.438304] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4091.923846] Lustre: server umount lustre-MDT0001 complete [ 4113.089654] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4115.941295] Key type lgssc unregistered [ 4116.281247] LNet: 145000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4116.285948] LNetError: 145000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4116.335437] LNet: Removed LNI 192.168.203.157@tcp [ 4117.260604] Key type .llcrypt unregistered [ 4117.263076] Key type ._llcrypt unregistered [ 4157.453815] Key type ._llcrypt registered [ 4157.456318] Key type .llcrypt registered [ 4157.550243] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 4171.896175] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4173.765409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4173.867422] alg: No test for adler32 (adler32-zlib) [ 4175.270856] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 4175.451599] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 4177.079173] Key type lgssc registered [ 4178.109862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4211.062903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4211.102506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.287839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4212.309892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4212.373676] Lustre: lustre-MDT0000: new disk, initializing [ 4212.453704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.465919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4216.505452] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4226.265475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.349391] Lustre: 149155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4226.470252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4226.474565] Lustre: Skipped 1 previous similar message [ 4226.567853] Lustre: lustre-MDT0001: new disk, initializing [ 4226.714779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.763838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4226.774369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4230.439915] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4240.075664] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4245.444329] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4252.335472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.877430] Lustre: lustre-OST0000: new disk, initializing [ 4252.901383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4252.913645] Lustre: 151122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.991785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.892589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4258.906572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4259.041680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4259.333887] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4270.070411] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4275.220860] Lustre: Failing over lustre-OST0000 [ 4275.340444] Lustre: server umount lustre-OST0000 complete [ 4293.157218] Lustre: 147380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170110/real 1786170110] req@0000000013e7aa05 x1872935002341888/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786170126 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4293.230169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.331678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.537302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.573862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.592385] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4295.555623] Lustre: 147378:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170112/real 1786170112] req@0000000082466301 x1872935002342144/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786170128 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4298.601858] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4300.968607] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4301.298110] Lustre: 147380:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170118/real 1786170118] req@000000008a419b08 x1872935002344064/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786170134 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4301.338421] Lustre: 147380:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4303.912699] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4303.913943] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4304.519896] Lustre: Failing over lustre-MDT0000 [ 4304.920705] Lustre: server umount lustre-MDT0000 complete [ 4306.591207] Lustre: 147379:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786170123/real 1786170123] req@00000000d3313697 x1872935002344960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786170139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4306.607808] Lustre: 147379:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4308.962090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.989478] Lustre: Skipped 2 previous similar messages [ 4314.081432] LustreError: 149166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.093603] LustreError: 149166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4319.205166] LustreError: 151130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.217273] LustreError: 151130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4323.163529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.242296] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4323.432149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.491548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4324.687036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4327.959673] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4328.434707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4328.445929] Lustre: Skipped 1 previous similar message [ 4328.460130] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4328.493527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4338.659420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.660113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.668426] Lustre: Skipped 1 previous similar message [ 4341.206066] Lustre: server umount lustre-OST0000 complete [ 4348.935848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.936263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4348.942681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.942690] Lustre: Skipped 1 previous similar message [ 4348.948037] Lustre: Skipped 1 previous similar message [ 4354.240410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.251932] Lustre: Skipped 2 previous similar messages [ 4358.372136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.376206] Lustre: Skipped 1 previous similar message [ 4360.159835] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4360.307426] Lustre: server umount lustre-MDT0000 complete [ 4363.750107] LustreError: 149161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.766297] LustreError: 149161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4365.466239] LustreError: 149147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170198 with bad export cookie 8366488398357846481 [ 4365.486534] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.488908] LustreError: 149147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4365.995272] Lustre: server umount lustre-MDT0001 complete [ 4378.075229] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4380.054690] Key type lgssc unregistered [ 4380.264871] LNet: 154781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4380.269665] LNetError: 154781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4380.278844] LNet: Removed LNI 192.168.203.157@tcp [ 4380.894120] Key type .llcrypt unregistered [ 4380.901378] Key type ._llcrypt unregistered [ 4417.611774] Key type ._llcrypt registered [ 4417.614929] Key type .llcrypt registered [ 4417.751984] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4419.144172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4419.202946] alg: No test for adler32 (adler32-zlib) [ 4420.255849] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 4420.583243] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 4422.426446] Key type lgssc registered [ 4423.340722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4431.014122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4431.024278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.666680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.437547] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4442.084538] LustreError: 156300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.114295] LustreError: 156300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4447.202187] LustreError: 156301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.514233] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4450.999479] LustreError: 156329:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4451.008995] LustreError: 156329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4451.017398] LustreError: 156329:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4451.355115] Lustre: server umount lustre-MDT0000 complete [ 4467.438465] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 4476.765423] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4510.705724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.889992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4510.916941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4511.087126] Lustre: lustre-MDT0000: new disk, initializing [ 4511.151940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.169645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4514.876321] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4525.546923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.680623] Lustre: 159633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4525.737275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4525.741321] Lustre: Skipped 1 previous similar message [ 4525.811391] Lustre: lustre-MDT0001: new disk, initializing [ 4525.893098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4525.966444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4526.001624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4530.327302] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4539.153209] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4545.147898] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4551.141048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.577669] Lustre: lustre-OST0000: new disk, initializing [ 4551.581953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4551.595616] Lustre: 161600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4551.747061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.299689] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4559.487184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4559.494411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4559.577198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4567.031873] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4571.451791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4573.682836] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:26:43 (1786170403) [ 4803.552793] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.560170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4803.560526] Lustre: Skipped 1 previous similar message [ 4803.568802] Lustre: Skipped 2 previous similar messages [ 4808.672702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.676058] Lustre: Skipped 2 previous similar messages [ 4809.580477] Lustre: server umount lustre-MDT0000 complete [ 4813.369422] LustreError: 159625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170646 with bad export cookie 14131178288317346071 [ 4813.373779] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.385674] LustreError: 159625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4813.791870] LustreError: 159639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4813.799794] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.828694] Lustre: Skipped 1 previous similar message [ 4813.832122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4818.915357] LustreError: 163387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.916329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4819.664090] Lustre: server umount lustre-MDT0001 complete [ 4829.551181] Lustre: server umount lustre-OST0000 complete [ 4837.360661] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 4847.134543] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4882.410202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.775521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4882.816576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4882.921722] Lustre: lustre-MDT0000: new disk, initializing [ 4883.033697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.059929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4888.056424] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4897.293262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4897.384268] Lustre: 166258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4897.422670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4897.426283] Lustre: Skipped 1 previous similar message [ 4897.479269] Lustre: lustre-MDT0001: new disk, initializing [ 4897.563295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4897.605483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4897.626424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4901.726556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4910.418976] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4916.206919] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4922.336517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4922.530926] Lustre: lustre-OST0000: new disk, initializing [ 4922.533921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4922.539078] Lustre: 168229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4922.614922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.037249] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4931.606572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4931.618113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4931.692036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4939.213066] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4944.035526] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4944.305789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4948.214575] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4948.463879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4952.036776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.048360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4952.064673] Lustre: Skipped 1 previous similar message [ 4956.405418] Lustre: server umount lustre-OST0000 complete [ 4962.273590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.277019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.279981] Lustre: Skipped 1 previous similar message [ 4962.289971] Lustre: Skipped 1 previous similar message [ 4967.395706] LustreError: 166265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.407692] LustreError: 166265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4967.421389] Lustre: server umount lustre-MDT0000 complete [ 4971.182259] LustreError: 168228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170804 with bad export cookie 14131178288317504901 [ 4971.190913] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.195969] LustreError: 168228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4971.665668] Lustre: server umount lustre-MDT0001 complete [ 4980.288462] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:33:30 (1786170810) [ 4995.238799] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 5005.043801] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5040.079200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.452452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5040.507743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5040.561813] Lustre: lustre-MDT0000: new disk, initializing [ 5040.632974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.649201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5044.685948] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5054.574042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.700745] Lustre: 172984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5054.946393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5060.290420] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5068.933759] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5074.128225] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5079.856187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.055617] Lustre: lustre-OST0000: new disk, initializing [ 5080.058576] Lustre: Skipped 1 previous similar message [ 5080.062643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5080.067031] Lustre: Skipped 2 previous similar messages [ 5080.074405] Lustre: 174962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5087.053680] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5087.829493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5087.838955] Lustre: Skipped 1 previous similar message [ 5087.850383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5087.967292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5096.826883] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5103.587701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.589695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.595108] LustreError: Skipped 1 previous similar message [ 5103.605462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5108.715730] LustreError: 174944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.735331] LustreError: 174944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5108.838451] Lustre: server umount lustre-OST0000 complete [ 5113.828598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.842043] Lustre: Skipped 2 previous similar messages [ 5118.398735] Lustre: server umount lustre-MDT0000 complete [ 5121.572303] LustreError: 173788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786170954 with bad export cookie 14131178288317505776 [ 5121.580142] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5121.582741] LustreError: 173788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5121.841352] Lustre: server umount lustre-MDT0001 complete [ 5130.670205] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:36:01 (1786170961) [ 5146.533315] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 5155.570722] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5192.584141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5192.867304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5192.916908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5193.021752] Lustre: lustre-MDT0000: new disk, initializing [ 5193.140059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.155300] Lustre: Skipped 2 previous similar messages [ 5193.167667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5199.318537] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5209.711301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.819590] Lustre: 179294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5210.280428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5214.552859] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5223.651464] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5230.750349] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5237.763713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.966606] Lustre: 181270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5243.392395] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5247.570776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5247.651718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5253.443921] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5263.335335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.339287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5263.340985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5263.340993] Lustre: Skipped 3 previous similar messages [ 5263.366160] Lustre: Skipped 1 previous similar message [ 5266.010804] Lustre: server umount lustre-OST0000 complete [ 5272.032245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5272.036250] LustreError: Skipped 1 previous similar message [ 5275.355939] Lustre: server umount lustre-MDT0000 complete [ 5278.690915] LustreError: 180107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.706871] LustreError: 180107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5278.856987] LustreError: 182376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171111 with bad export cookie 14131178288317506868 [ 5278.857823] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5278.865441] LustreError: 182376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5279.065897] Lustre: server umount lustre-MDT0001 complete [ 5289.887407] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5292.489407] Key type lgssc unregistered [ 5292.815739] LNet: 183132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5292.833709] LNetError: 183132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5293.861888] LNet: Removed LNI 192.168.203.157@tcp [ 5294.994620] Key type .llcrypt unregistered [ 5294.999095] Key type ._llcrypt unregistered [ 5320.712605] Key type ._llcrypt registered [ 5320.714740] Key type .llcrypt registered [ 5320.787428] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5322.528499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5322.540622] alg: No test for adler32 (adler32-zlib) [ 5323.546780] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5323.793328] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5325.559164] Key type lgssc registered [ 5326.716802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5333.791537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5333.812130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5335.156831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.259276] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5345.760755] LustreError: 184550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5345.779317] LustreError: 184550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5347.170990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.467990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5351.656508] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5359.587763] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5366.134776] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5372.797637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.052782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.787039] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5380.218484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5386.123768] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5408.224735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.230364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5408.231572] Lustre: Skipped 1 previous similar message [ 5408.237290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5408.241963] Lustre: Skipped 1 previous similar message [ 5413.382926] Lustre: server umount lustre-OST0000 complete [ 5418.465190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5418.466825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5418.487996] Lustre: Skipped 1 previous similar message [ 5418.503837] Lustre: Skipped 1 previous similar message [ 5423.586819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.628609] Lustre: Skipped 1 previous similar message [ 5428.707372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.711071] Lustre: Skipped 1 previous similar message [ 5431.263423] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5431.499573] Lustre: server umount lustre-MDT0000 complete [ 5433.827451] LustreError: 184550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.864967] LustreError: 184550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5435.276957] LustreError: 184532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171268 with bad export cookie 9563928978256387223 [ 5435.284713] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5435.288201] LustreError: 184532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5435.619348] Lustre: server umount lustre-MDT0001 complete [ 5446.434848] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5449.035791] Key type lgssc unregistered [ 5449.240677] LNet: 188108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5449.255248] LNetError: 188108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5450.344191] LNet: Removed LNI 192.168.203.157@tcp [ 5451.089610] Key type .llcrypt unregistered [ 5451.092613] Key type ._llcrypt unregistered [ 5474.438204] Key type ._llcrypt registered [ 5474.443830] Key type .llcrypt registered [ 5474.629201] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5476.187751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5476.477255] alg: No test for adler32 (adler32-zlib) [ 5477.686977] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 5477.880838] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5479.519806] Key type lgssc registered [ 5480.793392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5487.864348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5487.875865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.234366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.681346] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5499.361094] LustreError: 189527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.396963] LustreError: 189527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5502.171621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.433101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5507.229544] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5514.494956] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5520.570741] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5526.532552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5526.780906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.976868] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5533.937888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5539.668909] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5547.358845] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 5548.513226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5548.520074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.536420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.536814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5549.543222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.588050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.681183] Lustre: Skipped 1 previous similar message [ 5552.449779] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 5557.576336] Lustre: lustre-OST0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 5557.587103] Lustre: Skipped 2 previous similar messages [ 5561.311143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5561.473931] Lustre: server umount lustre-OST0000 complete [ 5569.954808] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e52882c8800.ost_server_uuid 50 [ 5571.670569] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5585.381417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.400790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.403729] Lustre: Skipped 1 previous similar message [ 5585.416785] Lustre: Skipped 3 previous similar messages [ 5591.175439] Lustre: server umount lustre-MDT0000 complete [ 5591.522365] LustreError: 190255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5591.538278] LustreError: 190255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5594.537800] LustreError: 189507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171427 with bad export cookie 12990189935601360891 [ 5594.547650] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5594.548455] LustreError: 189507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5594.866812] Lustre: server umount lustre-MDT0001 complete [ 5604.674676] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:43:54 (1786171434) [ 5611.282624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.747562] LustreError: 193444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.795851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.969956] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5617.124685] LustreError: 193445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.226380] LustreError: 193444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.168183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.388300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5627.709442] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5634.640530] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5642.032170] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5648.684747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5648.944051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.228206] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5656.120083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5661.540828] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5668.462587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5668.702370] Lustre: lustre-OST0001: new disk, initializing [ 5668.707619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5668.713721] Lustre: 196284:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5668.807027] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.413628] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5675.609057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5675.616502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5675.745303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5682.550403] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5686.252847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5686.268454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5686.276673] Lustre: Skipped 3 previous similar messages [ 5700.063468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5700.158465] Lustre: server umount lustre-OST0000 complete [ 5701.620348] LustreError: 195266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.653726] LustreError: 195266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5706.248340] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5706.587535] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5710.574678] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5710.834701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5719.355272] LustreError: 196699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5719.369235] LustreError: 196699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 5727.213248] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.216135] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.225593] LustreError: Skipped 1 previous similar message [ 5727.231257] Lustre: Skipped 1 previous similar message [ 5727.233914] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5727.241673] Lustre: Skipped 8 previous similar messages [ 5729.446872] Lustre: server umount lustre-OST0001 complete [ 5735.905316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5735.911255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5735.920552] Lustre: Skipped 1 previous similar message [ 5747.167191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5747.371826] Lustre: server umount lustre-MDT0000 complete [ 5750.741156] LustreError: 193425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171583 with bad export cookie 12990189935601361962 [ 5750.743615] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.749641] LustreError: 193425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5751.164824] Lustre: server umount lustre-MDT0001 complete [ 5775.831325] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:46:45 (1786171605) [ 5781.068851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.257642] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5781.297206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5781.422637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.703866] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5793.424263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.491236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5793.519768] Lustre: 199724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5798.146966] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5804.790264] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5810.170455] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5817.832347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.945096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5824.176500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5825.063702] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5832.168625] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5838.366421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.418817] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5844.490360] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5851.486808] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5855.204355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5855.224585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.230902] Lustre: Skipped 1 previous similar message [ 5855.234621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.237712] Lustre: Skipped 5 previous similar messages [ 5861.070253] Lustre: server umount lustre-OST0000 complete [ 5862.721345] LustreError: 201575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.730494] LustreError: 201575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5878.757665] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5883.015416] Lustre: server umount lustre-OST0001 complete [ 5889.508299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.531250] Lustre: Skipped 4 previous similar messages [ 5901.279662] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5901.554689] Lustre: server umount lustre-MDT0000 complete [ 5904.932722] LustreError: 199717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171738 with bad export cookie 12990189935601363369 [ 5904.943155] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.002832] LustreError: 199717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5905.367578] Lustre: server umount lustre-MDT0001 complete [ 5928.599335] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:49:18 (1786171758) [ 5943.563297] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 5953.670111] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5986.691772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5986.869496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5986.895589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5986.948991] Lustre: lustre-MDT0000: new disk, initializing [ 5987.001653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.010572] Lustre: Skipped 3 previous similar messages [ 5987.021232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5991.368916] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6001.999187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.095651] Lustre: 207834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6002.167940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6002.172196] Lustre: Skipped 1 previous similar message [ 6002.269175] Lustre: lustre-MDT0001: new disk, initializing [ 6002.485502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6002.496889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6007.081349] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6016.524747] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6022.319588] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6030.052142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.376759] Lustre: lustre-OST0000: new disk, initializing [ 6030.380988] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6030.386919] Lustre: 209804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6037.678752] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6039.734234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6039.760027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6040.041988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6047.817629] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6053.572899] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6053.725519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6058.680189] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6058.888271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6065.653455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.654171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.656298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.656304] Lustre: Skipped 12 previous similar messages [ 6065.668589] LustreError: Skipped 2 previous similar messages [ 6065.719101] Lustre: Skipped 1 previous similar message [ 6075.366783] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6075.494531] Lustre: server umount lustre-OST0000 complete [ 6086.495895] Lustre: server umount lustre-MDT0000 complete [ 6090.482883] LustreError: 207827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786171923 with bad export cookie 12990189935601364930 [ 6090.486741] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.491946] LustreError: 207827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6090.858234] Lustre: server umount lustre-MDT0001 complete [ 6099.107577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.526079] LustreError: 212022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.569696] LustreError: 212022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6104.438959] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6113.038689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.444768] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6125.738795] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6131.202420] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6137.623711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.072825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.084445] Lustre: Skipped 4 previous similar messages [ 6143.719134] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6150.536935] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6154.591895] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6154.792735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6159.363929] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6159.591001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6164.964537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.972053] LustreError: Skipped 1 previous similar message [ 6167.402629] Lustre: server umount lustre-OST0000 complete [ 6173.282657] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6177.580202] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6206.038269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.852981] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6219.496897] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6225.122803] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6230.921680] Lustre: server umount lustre-OST0000 complete [ 6236.128262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.135509] Lustre: Skipped 5 previous similar messages [ 6236.138333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6240.413313] Lustre: server umount lustre-MDT0000 complete [ 6241.248405] LustreError: 212019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.272934] LustreError: 212019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6244.169710] LustreError: 213867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172077 with bad export cookie 12990189935601365812 [ 6244.186841] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6244.188886] LustreError: 213867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6244.480858] Lustre: server umount lustre-MDT0001 complete [ 6253.945358] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:54:43 (1786172083) [ 6259.640845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.789591] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6271.883334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.885488] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6284.590587] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6290.204592] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6296.388223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.256866] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6308.241444] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6313.883185] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6314.102406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6319.277094] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6319.443504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6326.858279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.183861] Lustre: lustre-OST0001: new disk, initializing [ 6327.187189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6327.194277] Lustre: 221356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6332.740248] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6336.188140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6336.211881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6336.407768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6341.061845] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6346.044936] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6346.296840] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6350.582267] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6350.746893] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6356.974505] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6356.982401] Lustre: Skipped 14 previous similar messages [ 6358.308605] Lustre: server umount lustre-OST0001 complete [ 6363.954192] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6364.114750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6368.245571] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6368.492380] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6397.896646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6398.145500] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.161773] Lustre: Skipped 5 previous similar messages [ 6403.565295] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6403.586400] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6403.598606] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6403.613613] Lustre: Skipped 1 previous similar message [ 6403.993926] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6412.075438] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6413.695407] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6418.920390] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6418.934951] LustreError: Skipped 2 previous similar messages [ 6421.993661] Lustre: server umount lustre-OST0001 complete [ 6452.197614] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6452.339520] Lustre: server umount lustre-OST0000 complete [ 6461.852285] Lustre: server umount lustre-MDT0000 complete [ 6465.028311] LustreError: 221355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172298 with bad export cookie 12990189935601366764 [ 6465.039262] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.040973] LustreError: 221355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6479.328717] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6479.711940] Lustre: server umount lustre-MDT0001 complete [ 6501.545650] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:58:51 (1786172331) [ 6507.262926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.434393] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6507.463400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6511.883387] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6520.145692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.233463] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6520.256955] Lustre: 226604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6524.945344] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6533.628646] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.371561] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.412217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.717888] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6555.728821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6562.797081] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6572.913525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.005765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6579.926821] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6586.938844] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6588.555174] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6594.095631] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6594.385539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6599.136277] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6599.394448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6604.073361] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e52859e3000.ost_server_uuid 50 [ 6605.581541] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e52859e3000.ost_server_uuid in IDLE state after 0 sec [ 6606.649583] Lustre: Permanently deactivating lustre-OST0001 [ 6606.667891] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6609.531391] Lustre: Permanently reactivating lustre-OST0001 [ 6615.030747] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.050771] Lustre: Skipped 8 previous similar messages [ 6620.521595] Lustre: server umount lustre-OST0001 complete [ 6625.251017] LustreError: 229786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.261876] LustreError: 229786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 6630.107850] Lustre: server umount lustre-OST0000 complete [ 6648.287435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6648.518927] Lustre: server umount lustre-MDT0000 complete [ 6651.723584] LustreError: 229479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172484 with bad export cookie 12990189935601368052 [ 6651.733126] LustreError: 229479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6651.736606] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.074307] Lustre: server umount lustre-MDT0001 complete [ 6674.336224] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:01:44 (1786172504) [ 6681.814542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.026309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6682.056743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6682.064908] Lustre: Skipped 5 previous similar messages [ 6685.986513] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6695.783436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6695.849227] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6695.872804] Lustre: 233307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6700.398613] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6707.542610] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6713.111648] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.865262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6719.017746] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6719.032861] Lustre: Permanently deactivating lustre-OST0000 [ 6719.041912] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6725.008637] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6727.726708] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6731.351392] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6737.241886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.297541] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6743.044110] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6743.459973] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6749.766763] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6751.318874] Lustre: Permanently reactivating lustre-OST0000 [ 6751.336939] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6751.341264] Lustre: Skipped 2 previous similar messages [ 6760.665088] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 00000000e25e1892, cur 1786172593 deadline 1786172590 last 1786172560 [ 6761.488773] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6761.494837] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6761.505925] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6761.568109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6763.499657] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6763.550915] LustreError: Skipped 6 previous similar messages [ 6764.565636] Lustre: server umount lustre-OST0001 complete [ 6774.617918] Lustre: server umount lustre-OST0000 complete [ 6783.831154] Lustre: server umount lustre-MDT0000 complete [ 6787.368395] LustreError: 234059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172620 with bad export cookie 12990189935601369697 [ 6787.377969] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6787.381345] LustreError: 234059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6787.617586] Lustre: server umount lustre-MDT0001 complete [ 6799.222212] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6801.742777] Key type lgssc unregistered [ 6801.939760] LNet: 238169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6801.948351] LNetError: 238169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6801.958722] LNet: Removed LNI 192.168.203.157@tcp [ 6802.940727] Key type .llcrypt unregistered [ 6802.942746] Key type ._llcrypt unregistered [ 6825.035937] Key type ._llcrypt registered [ 6825.038620] Key type .llcrypt registered [ 6825.120473] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6826.243912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6826.395527] alg: No test for adler32 (adler32-zlib) [ 6827.528979] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 6827.726777] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6829.375176] Key type lgssc registered [ 6830.828823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6840.354021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6840.386585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6841.752963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6845.695981] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6851.107878] LustreError: 239637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.123758] LustreError: 239637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6853.469726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.563241] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6853.581495] Lustre: 239623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6853.591157] Lustre: Permanently deactivating lustre-MDT0001 [ 6853.607650] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6853.824057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6857.472407] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6857.487155] LustreError: 239667:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6857.501708] LustreError: 239667:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6857.511428] LustreError: 239667:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6858.167086] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6864.795701] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6869.678496] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6877.596447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.900622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6878.835799] LustreError: 241475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.566050] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6884.321986] LustreError: 241476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.505786] LustreError: 241476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.548366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6886.573489] LustreError: 241476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6890.449534] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6891.491280] LustreError: 241475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.530776] LustreError: 241475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6897.224728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.425221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6902.804713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6902.831210] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6903.169343] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6909.873325] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6913.632360] Lustre: Permanently reactivating lustre-MDT0001 [ 6913.643563] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6913.647341] Lustre: Skipped 2 previous similar messages [ 6920.033493] Lustre: lustre-MDT0001: Client b094bf07-b249-4164-ab0d-67c7f4ab5d85 (at 192.168.203.57@tcp) reconnecting [ 6925.938757] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6927.759757] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6933.962703] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6934.201431] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6938.218922] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6938.411255] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6940.617899] Lustre: Permanently deactivating lustre-MDT0001 [ 6940.629689] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6940.634110] Lustre: Skipped 2 previous similar messages [ 6947.374147] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6956.513925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6956.526388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.535393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6956.800786] Lustre: server umount lustre-MDT0000 complete [ 6959.074341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.078489] LustreError: 243305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6959.083360] Lustre: Skipped 1 previous similar message [ 6959.103833] LustreError: 243305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6960.297699] LustreError: 239617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786172793 with bad export cookie 14786305372126442275 [ 6960.303323] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.306839] LustreError: 239617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6960.733444] Lustre: server umount lustre-MDT0001 complete [ 6966.610889] Lustre: server umount lustre-OST0000 complete [ 6978.051593] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:06:48 (1786172808) [ 6987.559557] Lustre: server umount lustre-OST0001 complete [ 6995.425000] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 7004.014768] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7036.717287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.084961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7037.096252] Lustre: Skipped 2 previous similar messages [ 7037.115350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7037.211692] Lustre: lustre-MDT0000: new disk, initializing [ 7037.265571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.277629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7041.856239] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7052.132736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.273433] Lustre: 247894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7052.332502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7052.336347] Lustre: Skipped 1 previous similar message [ 7052.392244] Lustre: lustre-MDT0001: new disk, initializing [ 7052.463267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7052.494541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7052.520861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7056.822904] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7066.178864] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7070.683246] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.592191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.784964] Lustre: lustre-OST0000: new disk, initializing [ 7076.788487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7076.793662] Lustre: 249863:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7076.902740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.333327] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7085.179668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7085.184681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7085.511290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7093.664256] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.419572] LustreError: 248816:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7104.934876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.155511] Lustre: lustre-OST0001: new disk, initializing [ 7105.160852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7105.168517] Lustre: 250978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7106.447124] LustreError: 248816:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7106.816389] LustreError: 247903:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7111.630623] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7111.709694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7111.717821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7111.834131] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7116.831221] LustreError: 247903:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7122.091874] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7123.759931] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7126.372961] Lustre: lustre-OST0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 7127.023480] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7127.034882] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7127.076136] Lustre: Skipped 1 previous similar message [ 7131.454470] Lustre: lustre-OST0001: Not available for connect from 192.168.203.57@tcp (stopping) [ 7131.474095] Lustre: Skipped 2 previous similar messages [ 7131.778450] Lustre: server umount lustre-OST0001 complete [ 7132.289632] LustreError: 249844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7132.316661] LustreError: 249844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7152.097570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7152.152910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.164123] Lustre: Skipped 1 previous similar message [ 7152.170686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7152.181892] Lustre: Skipped 1 previous similar message [ 7158.244234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7158.247847] Lustre: Skipped 3 previous similar messages [ 7162.864553] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7163.003771] Lustre: server umount lustre-OST0000 complete [ 7168.481047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7168.484323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7168.493420] Lustre: Skipped 2 previous similar messages [ 7168.508724] Lustre: Skipped 1 previous similar message [ 7172.731403] Lustre: server umount lustre-MDT0000 complete [ 7173.606827] LustreError: 247906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7173.648199] LustreError: 247906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7176.031011] LustreError: 247887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173009 with bad export cookie 14786305372126444865 [ 7176.048521] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7176.066518] LustreError: 247887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7182.496397] Lustre: server umount lustre-MDT0001 complete [ 7192.461918] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7195.328672] Key type lgssc unregistered [ 7195.565491] LNet: 253017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7195.572730] LNetError: 253017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7195.587624] LNet: Removed LNI 192.168.203.157@tcp [ 7196.222889] Key type .llcrypt unregistered [ 7196.226445] Key type ._llcrypt unregistered [ 7234.742905] Key type ._llcrypt registered [ 7234.745197] Key type .llcrypt registered [ 7234.938553] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7236.351559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7236.370285] alg: No test for adler32 (adler32-zlib) [ 7237.538919] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7237.774430] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7239.431502] Key type lgssc registered [ 7240.382783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7247.387585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7247.403653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.683758] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7248.696517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7248.906759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.181629] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7261.906763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.965322] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7261.994799] Lustre: 255227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7262.479528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.137558] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7273.917329] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7280.829964] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7287.984603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7288.094400] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7288.273654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.466841] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7295.339058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7301.296806] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7356.908835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7356.916845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7356.950494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7358.741507] Lustre: server umount lustre-OST0000 complete [ 7363.573819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7375.384440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7375.694467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.792034] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7377.858377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7377.867544] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7382.127318] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7390.308690] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7429.615054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7429.620905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.630406] LustreError: Skipped 2 previous similar messages [ 7429.639557] Lustre: Skipped 1 previous similar message [ 7429.641662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7429.654411] Lustre: Skipped 2 previous similar messages [ 7434.211993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.216710] Lustre: Skipped 1 previous similar message [ 7435.523459] Lustre: server umount lustre-OST0000 complete [ 7441.891333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7441.899508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.917250] Lustre: Skipped 1 previous similar message [ 7441.927633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7444.465346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.406908] Lustre: server umount lustre-MDT0000 complete [ 7450.708893] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173283 with bad export cookie 13821244332599998472 [ 7450.710376] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7450.720228] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7450.987315] Lustre: server umount lustre-MDT0001 complete [ 7462.463712] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7464.746350] Key type lgssc unregistered [ 7465.136887] LNet: 260652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7465.142522] LNetError: 260652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7465.156687] LNet: Removed LNI 192.168.203.157@tcp [ 7466.131394] Key type .llcrypt unregistered [ 7466.133567] Key type ._llcrypt unregistered [ 7494.423681] Key type ._llcrypt registered [ 7494.425203] Key type .llcrypt registered [ 7494.522385] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7495.767440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7495.828258] alg: No test for adler32 (adler32-zlib) [ 7497.042249] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7497.240980] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7499.079262] Key type lgssc registered [ 7500.220718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7508.408327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7508.449924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.955797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7514.734563] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7520.239573] LustreError: 262068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.262627] LustreError: 262068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7523.080170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.435292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.052668] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7535.815708] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7541.610754] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7547.836339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.130381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7553.260605] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7555.330476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7560.537267] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7585.248444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7585.258642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.273359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.277984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7586.284621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7586.287322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.653965] Lustre: server umount lustre-OST0000 complete [ 7594.983300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7594.989765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.997410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7595.021600] Lustre: Skipped 1 previous similar message [ 7601.634432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.638438] Lustre: Skipped 2 previous similar messages [ 7606.753576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7606.758561] Lustre: Skipped 2 previous similar messages [ 7608.287140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7608.584289] Lustre: server umount lustre-MDT0000 complete [ 7611.455556] LustreError: 262049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173444 with bad export cookie 8839875539903710835 [ 7611.465396] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.470332] LustreError: 262049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7611.837380] Lustre: server umount lustre-MDT0001 complete [ 7621.560438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7624.770453] Key type lgssc unregistered [ 7625.056906] LNet: 266379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7625.065338] LNetError: 266379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7625.076993] LNet: Removed LNI 192.168.203.157@tcp [ 7625.976433] Key type .llcrypt unregistered [ 7625.979087] Key type ._llcrypt unregistered [ 7644.301867] Key type ._llcrypt registered [ 7644.304955] Key type .llcrypt registered [ 7644.443535] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7646.529499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7646.604112] alg: No test for adler32 (adler32-zlib) [ 7647.567172] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7647.730756] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7649.424658] Key type lgssc registered [ 7650.614398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7658.331132] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7658.417513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.924413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7664.636791] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7670.268745] LustreError: 267608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.284651] LustreError: 267608:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7673.290316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.544795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7677.430631] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7683.811536] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7690.321616] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7696.984532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.258199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7703.369817] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7705.790342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7710.785657] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7724.569354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7724.589094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7724.662968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.563260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7726.576100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.576528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7727.968374] Lustre: server umount lustre-OST0000 complete [ 7731.683038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.700891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7731.714834] Lustre: Skipped 1 previous similar message [ 7731.716184] Lustre: Skipped 1 previous similar message [ 7736.804786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.808626] Lustre: Skipped 2 previous similar messages [ 7737.161939] Lustre: server umount lustre-MDT0000 complete [ 7740.623974] LustreError: 267590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173573 with bad export cookie 14503550857706939629 [ 7740.649116] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.651272] LustreError: 267590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7741.090691] Lustre: server umount lustre-MDT0001 complete [ 7751.647183] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7754.605699] Key type lgssc unregistered [ 7754.810806] LNet: 271287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7754.815141] LNetError: 271287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7754.823965] LNet: Removed LNI 192.168.203.157@tcp [ 7755.500267] Key type .llcrypt unregistered [ 7755.503524] Key type ._llcrypt unregistered [ 7778.249503] Key type ._llcrypt registered [ 7778.251454] Key type .llcrypt registered [ 7778.401187] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7779.567185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7779.625975] alg: No test for adler32 (adler32-zlib) [ 7780.781855] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7781.086960] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7782.759212] Key type lgssc registered [ 7783.616959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7790.850360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7790.863251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.208474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.789970] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7802.341493] LustreError: 272684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.353852] LustreError: 272684:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7803.369966] LustreError: 272685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7803.550885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7803.823307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7808.113805] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7814.740655] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7819.953627] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7825.615752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.844650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7831.365479] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7837.731786] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7838.149419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7863.777171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7863.777438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.787670] LustreError: Skipped 1 previous similar message [ 7863.792098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.835164] Lustre: Skipped 1 previous similar message [ 7863.875040] Lustre: Skipped 1 previous similar message [ 7865.161942] Lustre: server umount lustre-OST0000 complete [ 7868.902108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.909596] Lustre: Skipped 1 previous similar message [ 7868.912883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.917125] Lustre: Skipped 1 previous similar message [ 7874.018132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.023435] Lustre: Skipped 1 previous similar message [ 7874.406418] Lustre: server umount lustre-MDT0000 complete [ 7877.327714] LustreError: 272665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173710 with bad export cookie 15879446995502215706 [ 7877.328974] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.334240] LustreError: 272665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7877.590071] Lustre: server umount lustre-MDT0001 complete [ 7886.954781] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7889.561235] Key type lgssc unregistered [ 7889.850934] LNet: 276991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7889.865802] LNetError: 276991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7889.883792] LNet: Removed LNI 192.168.203.157@tcp [ 7890.519179] Key type .llcrypt unregistered [ 7890.521994] Key type ._llcrypt unregistered [ 7907.588819] Key type ._llcrypt registered [ 7907.592720] Key type .llcrypt registered [ 7907.680893] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7908.762270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7908.907061] alg: No test for adler32 (adler32-zlib) [ 7910.231693] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 7910.544272] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7912.199179] Key type lgssc registered [ 7913.570547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7921.278156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7921.299559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.017248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7927.197806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7933.560144] LustreError: 278228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.593184] LustreError: 278228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7935.442767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.053395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7941.255324] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7948.402615] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7954.895767] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7962.417391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.648114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.007549] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7968.919415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7974.809161] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7989.733930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7989.738926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.745786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7991.780176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7991.784685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7991.800056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.813143] Lustre: Skipped 1 previous similar message [ 7995.903905] Lustre: server umount lustre-OST0000 complete [ 8002.017675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.023965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8002.036171] Lustre: Skipped 1 previous similar message [ 8002.047512] Lustre: Skipped 1 previous similar message [ 8005.664143] Lustre: server umount lustre-MDT0000 complete [ 8007.156170] LustreError: 278215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8007.187142] LustreError: 278215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8009.028546] LustreError: 278193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786173842 with bad export cookie 14719698149499428310 [ 8009.047486] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8009.049930] LustreError: 278193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8009.457380] Lustre: server umount lustre-MDT0001 complete [ 8020.811394] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 8023.073227] Key type lgssc unregistered [ 8023.442847] LNet: 281935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8023.447902] LNetError: 281935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8024.484679] LNet: Removed LNI 192.168.203.157@tcp [ 8025.371491] Key type .llcrypt unregistered [ 8025.374550] Key type ._llcrypt unregistered [ 8038.477751] Key type ._llcrypt registered [ 8038.480499] Key type .llcrypt registered [ 8038.696621] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8057.063695] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 8070.827185] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8073.081707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8073.138241] alg: No test for adler32 (adler32-zlib) [ 8074.114214] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 8074.572670] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 8076.287175] Key type lgssc registered [ 8077.240535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8111.656909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8111.681044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.263360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8113.285660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8113.358709] Lustre: lustre-MDT0000: new disk, initializing [ 8113.412765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.434553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8117.409127] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8127.220992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.305864] Lustre: 285487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8127.347290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8127.351153] Lustre: Skipped 1 previous similar message [ 8127.403777] Lustre: lustre-MDT0001: new disk, initializing [ 8127.487275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8127.530578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8127.557613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8131.696467] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8141.449497] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8147.201887] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8153.832377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.228887] Lustre: lustre-OST0000: new disk, initializing [ 8154.234330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8154.241864] Lustre: 287456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8154.411379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.822850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8159.831407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8159.951543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8161.166274] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8169.648385] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.190494] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8175.371658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8179.947319] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8180.118190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8185.827176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8185.832155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8185.852437] LustreError: Skipped 1 previous similar message [ 8185.872865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8188.143647] Lustre: server umount lustre-OST0000 complete [ 8194.017906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8194.027564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.066891] Lustre: Skipped 1 previous similar message [ 8194.087461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8194.092979] Lustre: Skipped 1 previous similar message [ 8196.066617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8196.071573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.996514] Lustre: server umount lustre-MDT0000 complete [ 8202.145335] LustreError: 285479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174035 with bad export cookie 16723555779893772559 [ 8202.156204] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8202.158165] LustreError: 285479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8202.373883] Lustre: server umount lustre-MDT0001 complete [ 8210.034912] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:27:20 (1786174040) [ 8215.204226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.480486] LustreError: 289865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.502121] LustreError: 289865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8215.558108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8219.758819] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8220.655207] LustreError: 289866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.761432] LustreError: 289865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.398128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.608097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.066422] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8236.749979] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8243.171327] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8250.800096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.246035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.685345] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8265.291664] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8272.867643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8272.877147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.889646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.897768] Lustre: Skipped 1 previous similar message [ 8275.936177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8278.767166] Lustre: server umount lustre-OST0000 complete [ 8283.106494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8283.110079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8283.125223] Lustre: Skipped 1 previous similar message [ 8283.130565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8283.135680] Lustre: Skipped 2 previous similar messages [ 8289.002443] Lustre: server umount lustre-MDT0000 complete [ 8291.296414] LustreError: 289866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8291.322756] LustreError: 289866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8293.058098] LustreError: 289847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174126 with bad export cookie 16723555779893773441 [ 8293.066397] LustreError: 289847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8293.071516] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8293.308871] Lustre: server umount lustre-MDT0001 complete [ 8304.437801] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 8306.574670] Key type lgssc unregistered [ 8307.285537] LNet: 293405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8307.292982] LNetError: 293405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8308.326196] LNet: Removed LNI 192.168.203.157@tcp [ 8309.103786] Key type .llcrypt unregistered [ 8309.105890] Key type ._llcrypt unregistered [ 8338.644518] Key type ._llcrypt registered [ 8338.646821] Key type .llcrypt registered [ 8338.774681] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 8354.376474] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8356.500717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8356.524091] alg: No test for adler32 (adler32-zlib) [ 8358.220504] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 8358.592564] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 8360.250961] Key type lgssc registered [ 8361.792948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8408.463783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8408.479265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.719077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8409.754273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8409.831266] Lustre: lustre-MDT0000: new disk, initializing [ 8409.915250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.926669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8414.488731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8424.611437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.718709] Lustre: 297234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8424.755406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8424.758650] Lustre: Skipped 1 previous similar message [ 8424.808479] Lustre: lustre-MDT0001: new disk, initializing [ 8424.878758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8424.924193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8424.941799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8429.381546] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8438.744704] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8445.517826] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8452.826937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8453.008826] Lustre: lustre-OST2710: new disk, initializing [ 8453.012342] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8453.019818] Lustre: 299203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8453.099699] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8459.730677] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8463.034359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8463.040721] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8463.227026] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8469.021402] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8475.254377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.432749] Lustre: lustre-OST03e8: new disk, initializing [ 8475.436315] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8475.441310] Lustre: 300265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8475.583548] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8481.533410] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8484.451618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8484.469188] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8484.630424] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8489.232454] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8496.006489] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8496.148956] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8500.343863] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8500.577127] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8505.905987] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8506.448648] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8511.747120] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8512.340747] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8517.088598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8517.092773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.100751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.688131] Lustre: server umount lustre-MDT0000 complete [ 8521.201182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8521.208563] Lustre: Skipped 1 previous similar message [ 8522.372595] LustreError: 297227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174355 with bad export cookie 13519441646923795962 [ 8522.389212] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8523.390516] Lustre: server umount lustre-MDT0001 complete [ 8536.142806] Lustre: server umount lustre-OST2710 complete [ 8550.329827] Lustre: server umount lustre-OST03e8 complete [ 8567.699634] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 8580.533551] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8616.221156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.408577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8616.428650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8616.491974] Lustre: lustre-MDT0000: new disk, initializing [ 8616.551473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8616.578682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8620.941811] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8632.585285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.665317] Lustre: 305516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8632.703741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8632.707117] Lustre: Skipped 1 previous similar message [ 8632.758180] Lustre: lustre-MDT0001: new disk, initializing [ 8632.852448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8632.886599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8632.899268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8638.887911] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8649.340728] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8655.167339] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8661.436950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8661.745348] Lustre: lustre-OST0000: new disk, initializing [ 8661.748928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8661.755807] Lustre: 307484:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8661.849807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.948904] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8668.063588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8668.111054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8668.321323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8677.550162] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8682.705552] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8682.998287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8687.599198] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8687.963666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8693.746949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.748871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.761053] Lustre: Skipped 2 previous similar messages [ 8696.181593] Lustre: server umount lustre-OST0000 complete [ 8703.974542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8703.991073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8704.018319] Lustre: Skipped 1 previous similar message [ 8704.020758] Lustre: Skipped 1 previous similar message [ 8706.635587] Lustre: server umount lustre-MDT0000 complete [ 8709.090053] LustreError: 305523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8709.121820] LustreError: 305523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8710.005038] LustreError: 305508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174543 with bad export cookie 13519441646923797642 [ 8710.014232] LustreError: 305508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8710.014627] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8710.317371] Lustre: server umount lustre-MDT0001 complete [ 8718.625000] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:35:48 (1786174548) [ 8720.554194] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8722.942189] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:35:52 (1786174552) [ 8726.156579] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8749.948797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.237213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8750.297881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8750.450864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.121928] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8763.945054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.005360] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8764.023552] Lustre: 311342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8769.495344] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8791.714241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8791.816571] Lustre: Denying initial registration attempt from nid 192.168.203.157@tcp, specified as failover [ 8791.822284] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8791.826993] LustreError: 312795:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8791.830650] LustreError: 312795:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8791.835934] LustreError: 312795:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8791.891472] Lustre: server umount lustre-OST0000 complete [ 8791.893758] LustreError: 312795:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8800.224691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8800.230367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.239088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8800.245300] Lustre: Skipped 1 previous similar message [ 8805.267853] Lustre: server umount lustre-MDT0000 complete [ 8805.861110] LustreError: 312123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.898670] LustreError: 312123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8809.073517] LustreError: 312093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174642 with bad export cookie 13519441646923798524 [ 8809.075936] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8809.079139] LustreError: 312093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8809.283631] Lustre: server umount lustre-MDT0001 complete [ 8821.176593] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 8823.759825] Key type lgssc unregistered [ 8823.965545] LNet: 313920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8823.969857] LNetError: 313920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8823.978249] LNet: Removed LNI 192.168.203.157@tcp [ 8825.073812] Key type .llcrypt unregistered [ 8825.077633] Key type ._llcrypt unregistered [ 8837.269184] Key type ._llcrypt registered [ 8837.273750] Key type .llcrypt registered [ 8837.373986] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8838.412126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8838.468134] alg: No test for adler32 (adler32-zlib) [ 8839.714797] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 8839.937630] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 8841.575213] Key type lgssc registered [ 8842.709936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8888.172438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 8897.909493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8897.925611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.148418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8899.168693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8899.284109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.196573] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8912.935082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8912.992532] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8913.009686] Lustre: 316687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8913.202745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8916.817388] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8924.063974] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8929.394471] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8948.772212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8948.930272] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8949.080291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.508780] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 8961.728232] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8966.627502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.633630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8966.637113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8966.671284] LustreError: Skipped 1 previous similar message [ 8969.833979] Lustre: server umount lustre-OST0000 complete [ 8974.816522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8974.820162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.826634] Lustre: Skipped 1 previous similar message [ 8974.830174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8974.833761] Lustre: Skipped 1 previous similar message [ 8976.864660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8976.873757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8978.876941] Lustre: server umount lustre-MDT0000 complete [ 8982.761743] LustreError: 316679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174815 with bad export cookie 17906897018977996630 [ 8982.789264] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.811195] LustreError: 316679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8983.217583] Lustre: server umount lustre-MDT0001 complete [ 8992.881434] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 8995.746362] Key type lgssc unregistered [ 8995.967862] LNet: 320261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8995.980971] LNetError: 320261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8995.994860] LNet: Removed LNI 192.168.203.157@tcp [ 8996.567720] Key type .llcrypt unregistered [ 8996.571302] Key type ._llcrypt unregistered [ 9019.597302] Key type ._llcrypt registered [ 9019.599646] Key type .llcrypt registered [ 9019.682377] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 9020.835869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9020.905646] alg: No test for adler32 (adler32-zlib) [ 9021.954990] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 9022.429246] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 9024.064197] Key type lgssc registered [ 9025.439850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9032.669718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9032.686233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.130424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9038.063501] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9044.455412] LustreError: 321677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.479468] LustreError: 321677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9046.565584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.844341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9050.860914] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9057.826525] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9063.021724] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9072.895427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9073.209224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.228222] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9085.562937] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9102.307035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.316066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.326859] Lustre: Skipped 2 previous similar messages [ 9102.329057] Lustre: Skipped 1 previous similar message [ 9103.740028] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 9105.310926] Lustre: server umount lustre-MDT0000 complete [ 9107.435364] LustreError: 322733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.451096] LustreError: 322733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9108.802157] LustreError: 321657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786174941 with bad export cookie 6020448410857006102 [ 9108.805713] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9108.811336] LustreError: 321657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9108.832439] LustreError: 321672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.554280] LustreError: 322733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9112.555720] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9112.595201] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9118.724650] LustreError: 321674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9118.724869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9118.778701] LustreError: 321674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9118.800925] Lustre: Skipped 2 previous similar messages [ 9123.307744] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9123.666303] Lustre: server umount lustre-MDT0001 complete [ 9129.249058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9137.208037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9153.567607] LustreError: 320986:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000349f1c21 x1872940084798976/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9153.799951] LustreError: 323511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9153.814391] LustreError: 323511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9153.880360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9153.938979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9158.144348] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9164.995372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9165.339724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9169.669796] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9173.479408] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9175.707295] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9177.527970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9183.057376] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9184.891163] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9188.833370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9188.836940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9188.881618] Lustre: Skipped 1 previous similar message [ 9188.889568] Lustre: Skipped 2 previous similar messages [ 9199.074831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9199.087618] Lustre: Skipped 3 previous similar messages [ 9203.168217] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9203.275408] Lustre: server umount lustre-OST0000 complete [ 9209.314799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.334546] Lustre: Skipped 1 previous similar message [ 9214.148062] Lustre: server umount lustre-MDT0000 complete [ 9214.435130] LustreError: 325299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9214.472599] LustreError: 325299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9217.716020] LustreError: 321657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175050 with bad export cookie 6020448410857013711 [ 9217.739897] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9217.747072] LustreError: 321657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9218.017621] Lustre: server umount lustre-MDT0001 complete [ 9230.550567] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 9232.763183] Key type lgssc unregistered [ 9233.106914] LNet: 327971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9233.114565] LNetError: 327971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9233.189248] LNet: Removed LNI 192.168.203.157@tcp [ 9234.015905] Key type .llcrypt unregistered [ 9234.018117] Key type ._llcrypt unregistered [ 9257.084167] Key type ._llcrypt registered [ 9257.086295] Key type .llcrypt registered [ 9257.171711] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 9258.717467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9258.820411] alg: No test for adler32 (adler32-zlib) [ 9259.834774] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 9260.063788] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 9261.751308] Key type lgssc registered [ 9263.263250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9270.133295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9270.146579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.429192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9275.764198] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9281.092832] LustreError: 329388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.127099] LustreError: 329388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9283.598553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9283.840572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9287.683932] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9295.436700] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9301.018575] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9304.544188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9304.552819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.574885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9307.109140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.109319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9307.146387] Lustre: Skipped 1 previous similar message [ 9308.193497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.199620] Lustre: Skipped 1 previous similar message [ 9310.366772] Lustre: server umount lustre-MDT0000 complete [ 9313.250700] LustreError: 329388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9313.270548] LustreError: 329388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9314.683030] LustreError: 329369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175147 with bad export cookie 14728605519197916208 [ 9314.700898] LustreError: 329369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9314.705413] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9314.965515] Lustre: server umount lustre-MDT0001 complete [ 9326.848442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.042324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9327.070379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9327.216221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.943171] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9338.228264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.294374] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9338.313431] Lustre: 331980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9338.606698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9342.672988] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9348.692510] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9353.700657] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9362.092147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9362.249865] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9362.262060] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9362.271960] LustreError: 333861:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9362.278785] LustreError: 333861:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9362.284044] LustreError: 333861:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9362.333844] Lustre: server umount lustre-OST0000 complete [ 9362.336485] LustreError: 333861:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9371.367018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9371.478613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9371.647422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.306788] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9381.494422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9383.289694] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9391.214760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.322793] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9391.397856] Lustre: lustre-OST0001: new disk, initializing [ 9391.401528] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9391.407914] Lustre: 335402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9391.490556] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9391.585395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9391.598350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9391.629517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 9397.790251] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9404.062531] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9406.944858] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9406.951845] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9406.955209] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9406.960110] Lustre: Skipped 1 previous similar message [ 9412.072525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9412.079563] Lustre: Skipped 2 previous similar messages [ 9412.494849] Lustre: server umount lustre-OST0001 complete [ 9417.188665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9417.190442] LustreError: 334327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.194942] Lustre: Skipped 1 previous similar message [ 9417.216690] LustreError: 334327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9421.656954] Lustre: server umount lustre-OST0000 complete [ 9425.961114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9425.988602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9426.059574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9426.062974] Lustre: Skipped 2 previous similar messages [ 9431.431477] Lustre: server umount lustre-MDT0000 complete [ 9432.550171] LustreError: 331986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.566233] LustreError: 331986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9435.180689] LustreError: 331971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175268 with bad export cookie 14728605519197916761 [ 9435.189571] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9435.196360] LustreError: 331971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9435.656990] Lustre: server umount lustre-MDT0001 complete [ 9446.838807] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 9449.244132] Key type lgssc unregistered [ 9449.551719] LNet: 337364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9449.557760] LNetError: 337364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9449.566799] LNet: Removed LNI 192.168.203.157@tcp [ 9450.211757] Key type .llcrypt unregistered [ 9450.213722] Key type ._llcrypt unregistered [ 9516.639704] Key type ._llcrypt registered [ 9516.641883] Key type .llcrypt registered [ 9516.731511] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 9531.436311] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 9532.238641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9532.485984] alg: No test for adler32 (adler32-zlib) [ 9533.833419] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 9534.131338] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 9535.866194] Key type lgssc registered [ 9537.111714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9569.840125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9569.896651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.189303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9571.225091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9571.280261] Lustre: lustre-MDT0000: new disk, initializing [ 9571.330262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9571.344121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9575.900428] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9585.300818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.420975] Lustre: 342297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9585.496203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9585.500425] Lustre: Skipped 1 previous similar message [ 9585.560785] Lustre: lustre-MDT0001: new disk, initializing [ 9585.759698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.902186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9585.915757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9590.622491] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9599.278471] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9604.621295] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9610.640060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.814793] Lustre: lustre-OST0000: new disk, initializing [ 9610.819714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9610.829338] Lustre: 344265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9610.923422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.456268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9616.464797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9616.626621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9617.002541] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9625.314369] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9629.564670] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9630.026442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9633.987281] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9634.207957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9637.374845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9637.380218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.437366] LustreError: Skipped 1 previous similar message [ 9637.443454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9637.453538] Lustre: Skipped 1 previous similar message [ 9637.493621] Lustre: Skipped 1 previous similar message [ 9642.476792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9642.483306] Lustre: Skipped 1 previous similar message [ 9642.634182] Lustre: server umount lustre-OST0000 complete [ 9647.072627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9647.086472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.102229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.713614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9652.718173] Lustre: Skipped 2 previous similar messages [ 9657.825376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.830275] Lustre: Skipped 1 previous similar message [ 9660.383329] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9660.491391] Lustre: server umount lustre-MDT0000 complete [ 9664.199653] LustreError: 342290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175497 with bad export cookie 9161287014368786325 [ 9664.204779] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.215809] LustreError: 342290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9664.591918] Lustre: server umount lustre-MDT0001 complete [ 9672.599486] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:51:42 (1786175502) [ 9679.771559] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:51:50 (1786175510) [ 9687.781805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.515271] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.531176] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9688.582057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.702847] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9693.679520] LustreError: 347025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9698.787780] LustreError: 347024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9701.027279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.398515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9705.629643] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9712.283836] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9718.042955] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.008331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.387515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.854717] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9737.637549] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9745.446335] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9747.248826] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9748.819727] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9750.662835] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9752.381026] Lustre: Failing over lustre-MDT0000 [ 9752.546473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9752.565194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9752.577157] Lustre: Skipped 1 previous similar message [ 9752.581757] LustreError: 347752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.593940] LustreError: 347752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9752.630507] Lustre: server umount lustre-MDT0000 complete [ 9756.132356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.150617] Lustre: Skipped 1 previous similar message [ 9760.576507] LustreError: 348043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9760.615613] LustreError: 348043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9770.839355] LustreError: 347020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9770.865048] LustreError: 347020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9771.719655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9771.804826] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9772.193873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9772.330407] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9777.469715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9777.631328] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9777.644116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9777.854860] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9777.913108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9786.049145] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9787.406450] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9791.339743] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9793.435868] Lustre: Failing over lustre-MDT0000 [ 9793.506051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9793.510617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9793.518315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9793.626900] Lustre: server umount lustre-MDT0000 complete [ 9796.067419] LustreError: 349465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.084325] LustreError: 349465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9812.319305] Lustre: 340522:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786175629/real 1786175629] req@00000000e9aa8451 x1872940621714304/t0(0) o400->MGC192.168.203.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786175645 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9812.380883] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.787575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9822.694782] LustreError: 340521:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec9b7bea x1872940621720320/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9823.006149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.053188] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9824.581751] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9827.808544] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9828.344917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9828.354385] Lustre: Skipped 2 previous similar messages [ 9828.376042] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9828.465221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9837.300466] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9838.808423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9848.803066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9848.803908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9848.810518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9848.813965] Lustre: Skipped 3 previous similar messages [ 9851.189994] Lustre: server umount lustre-MDT0000 complete [ 9853.922979] LustreError: 347025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9853.981611] LustreError: 347025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [ 9854.694435] LustreError: 347006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175687 with bad export cookie 9161287014368789034 [ 9854.715438] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9854.928092] Lustre: server umount lustre-MDT0001 complete [ 9868.706653] Lustre: server umount lustre-OST0000 complete [ 9885.539395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9889.358355] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9897.752418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.014117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9898.021366] Lustre: Skipped 1 previous similar message [ 9905.891749] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9914.556179] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9919.752430] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9926.140359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9930.667123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9933.279922] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 9941.051422] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9951.201329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9951.212936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.227376] Lustre: Skipped 1 previous similar message [ 9951.236379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9951.245695] Lustre: Skipped 2 previous similar messages [ 9956.594186] Lustre: server umount lustre-OST0000 complete [ 9966.446240] Lustre: server umount lustre-MDT0000 complete [ 9968.609287] LustreError: 353363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.622191] LustreError: 353363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9970.188463] LustreError: 356144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175803 with bad export cookie 9161287014368789783 [ 9970.221808] LustreError: 356144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9970.226708] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9971.033986] Lustre: server umount lustre-MDT0001 complete [ 9979.715712] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 9982.328615] Key type lgssc unregistered [ 9982.610367] LNet: 356898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9982.622943] LNetError: 356898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.656322] LNet: Removed LNI 192.168.203.157@tcp [ 9984.487764] Key type .llcrypt unregistered [ 9984.489697] Key type ._llcrypt unregistered [10008.266325] Key type ._llcrypt registered [10008.268416] Key type .llcrypt registered [10008.383489] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [10010.184244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10010.200536] alg: No test for adler32 (adler32-zlib) [10011.262111] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [10011.449925] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [10013.143295] Key type lgssc registered [10014.203382] Lustre: Echo OBD driver; http://www.lustre.org/ [10020.199340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10020.232182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.536276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.604034] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10031.074635] LustreError: 358397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10031.083151] LustreError: 358397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10032.991846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.250818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10037.976625] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10045.268511] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10050.755318] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10056.884682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10057.121233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10063.026976] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10066.437491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10070.223649] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.786739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10081.791744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.810952] Lustre: Skipped 2 previous similar messages [10081.827700] Lustre: Skipped 2 previous similar messages [10083.671404] Lustre: server umount lustre-MDT0000 complete [10086.881548] LustreError: 359386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.892908] LustreError: 359386:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10087.029732] LustreError: 358377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786175920 with bad export cookie 700800535874131555 [10087.035655] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.040885] LustreError: 358377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10087.342700] Lustre: server umount lustre-MDT0001 complete [10100.681766] Lustre: server umount lustre-OST0000 complete [10115.154622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10115.437608] LustreError: 362106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10115.529810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10119.660646] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10120.678439] LustreError: 362107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10125.796641] LustreError: 362106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.984746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.551917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10132.731731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10140.759263] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10146.161625] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10151.856627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10152.096573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.657464] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10160.354574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10164.316545] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10175.970420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10175.979286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10175.981274] Lustre: Skipped 1 previous similar message [10175.989130] Lustre: Skipped 2 previous similar messages [10181.092099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10181.095540] Lustre: Skipped 2 previous similar messages [10181.874234] Lustre: server umount lustre-MDT0000 complete [10185.372659] LustreError: 362088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176018 with bad export cookie 700800535874132668 [10185.373144] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.379648] LustreError: 362088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10185.644683] Lustre: server umount lustre-MDT0001 complete [10200.906846] Lustre: server umount lustre-OST0000 complete [10216.535384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10216.909727] LustreError: 366017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.967654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10221.702595] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10230.513782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.082328] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10243.754762] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10249.572605] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10256.871652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.149630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10257.160656] Lustre: Skipped 1 previous similar message [10263.130121] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10263.294621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10270.359381] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10273.762806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.763512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10273.777968] Lustre: Skipped 1 previous similar message [10273.780941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10279.945529] Lustre: server umount lustre-OST0000 complete [10286.048889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.052862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.056304] Lustre: Skipped 1 previous similar message [10286.082635] Lustre: Skipped 3 previous similar messages [10289.382357] Lustre: server umount lustre-MDT0000 complete [10291.173406] LustreError: 366018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10291.211470] LustreError: 366018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10292.936255] LustreError: 368808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176126 with bad export cookie 700800535874133690 [10292.951374] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10292.951872] LustreError: 368808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10293.368610] Lustre: server umount lustre-MDT0001 complete [10304.042663] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [10306.138938] Key type lgssc unregistered [10306.395689] LNet: 369562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10306.418525] LNetError: 369562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10306.437744] LNet: Removed LNI 192.168.203.157@tcp [10307.200609] Key type .llcrypt unregistered [10307.204578] Key type ._llcrypt unregistered [10334.856368] Key type ._llcrypt registered [10334.858858] Key type .llcrypt registered [10335.117496] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [10336.674295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10336.778392] alg: No test for adler32 (adler32-zlib) [10338.129722] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [10338.341462] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [10340.224678] Key type lgssc registered [10341.858854] Lustre: Echo OBD driver; http://www.lustre.org/ [10349.718468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10349.728347] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10349.734551] LustreError: 371005:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10349.747453] LustreError: 371005:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10349.752307] LustreError: 371005:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10349.757429] LustreError: 371005:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10349.762850] LustreError: 371005:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10356.823226] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10356.828248] LustreError: 371402:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10356.840419] LustreError: 371402:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10356.845388] LustreError: 371402:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10356.850531] LustreError: 371402:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10356.855948] LustreError: 371402:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10374.017196] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [10376.383836] Key type lgssc unregistered [10376.757902] LNet: 372408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10376.762628] LNetError: 372408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10376.773941] LNet: Removed LNI 192.168.203.157@tcp [10377.625588] Key type .llcrypt unregistered [10377.635716] Key type ._llcrypt unregistered [10394.056782] Key type ._llcrypt registered [10394.059147] Key type .llcrypt registered [10394.157090] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [10409.730397] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [10410.909203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10410.929210] alg: No test for adler32 (adler32-zlib) [10411.970472] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [10412.188174] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [10413.855169] Key type lgssc registered [10414.823304] Lustre: Echo OBD driver; http://www.lustre.org/ [10447.924387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10447.966223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10449.158275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10449.180759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10449.236283] Lustre: lustre-MDT0000: new disk, initializing [10449.310233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10449.327895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10454.324851] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10463.125333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.253526] Lustre: 375564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10463.309524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10463.320222] Lustre: Skipped 1 previous similar message [10463.393242] Lustre: lustre-MDT0001: new disk, initializing [10463.498412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10463.518786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10463.529800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10467.165536] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10476.033541] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10481.718580] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10488.094420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.472901] Lustre: lustre-OST0000: new disk, initializing [10488.475457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10488.575175] Lustre: 377536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10488.790750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10495.046960] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10498.107831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10498.119506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10498.220036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10502.405252] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10506.140699] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10506.309147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10511.030202] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10511.796483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10513.896679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.897821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10513.911584] Lustre: Skipped 1 previous similar message [10513.916214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10513.959906] LustreError: Skipped 1 previous similar message [10519.013817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10519.022654] Lustre: Skipped 2 previous similar messages [10519.674572] Lustre: server umount lustre-OST0000 complete [10524.136808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.162965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10524.174483] Lustre: Skipped 1 previous similar message [10529.200776] Lustre: server umount lustre-MDT0000 complete [10532.613739] LustreError: 375557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176365 with bad export cookie 6175789948310405878 [10532.617293] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10532.624703] LustreError: 375557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10532.880354] Lustre: server umount lustre-MDT0001 complete [10542.115293] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10543.693527] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:06:13 (1786176373) [10550.895889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.150354] LustreError: 379993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10551.164541] LustreError: 379993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10551.219563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10555.292723] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10556.389499] LustreError: 379994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10561.504147] LustreError: 379993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.868732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.098959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10569.017949] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10575.970344] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10580.800920] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10587.652581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.884923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10594.611156] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10602.318892] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10608.995606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10609.230875] Lustre: lustre-OST0001: new disk, initializing [10609.234119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10609.241339] Lustre: 382828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10609.315186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10615.401090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10615.411848] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10615.499607] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10615.541852] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10624.543579] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10631.145171] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.152847] Lustre: Skipped 1 previous similar message [10631.153228] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10631.156462] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10631.165788] LustreError: Skipped 1 previous similar message [10635.470067] Lustre: server umount lustre-OST0001 complete [10636.268072] LustreError: 381816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.302612] LustreError: 381816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10641.382202] LustreError: 381816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.399948] LustreError: 381816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10651.617912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10651.621216] LustreError: 382836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.621494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10651.621497] Lustre: Skipped 2 previous similar messages [10651.632852] Lustre: Skipped 2 previous similar messages [10651.669856] LustreError: 382836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10657.826770] Lustre: server umount lustre-OST0000 complete [10661.860525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10661.860794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10661.863535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10661.863542] Lustre: Skipped 3 previous similar messages [10662.057283] Lustre: Skipped 1 previous similar message [10675.167308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10675.375443] Lustre: server umount lustre-MDT0000 complete [10677.218206] LustreError: 379993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.246679] LustreError: 379993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10678.319687] LustreError: 382827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176511 with bad export cookie 6175789948310406760 [10678.325597] LustreError: 382827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10678.325672] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10684.673314] Lustre: server umount lustre-MDT0001 complete [10694.159804] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [10696.742149] Key type lgssc unregistered [10697.108257] LNet: 384820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10697.116974] LNetError: 384820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10697.135284] LNet: Removed LNI 192.168.203.157@tcp [10697.809850] Key type .llcrypt unregistered [10697.811811] Key type ._llcrypt unregistered [10743.226314] Key type ._llcrypt registered [10743.228876] Key type .llcrypt registered [10743.316860] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [10744.448575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10744.465650] alg: No test for adler32 (adler32-zlib) [10745.835571] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [10746.152158] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [10747.775182] Key type lgssc registered [10749.207092] Lustre: Echo OBD driver; http://www.lustre.org/ [10756.491441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10756.514392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10757.728918] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10757.743217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10757.865973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10761.737158] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10769.008554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.078574] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10769.100501] Lustre: 387618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10769.311700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10773.098516] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10779.625409] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10784.805477] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10789.860205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10789.875083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10789.890236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10790.882267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10790.884814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.830934] Lustre: server umount lustre-MDT0000 complete [10795.618618] LustreError: 387612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176628 with bad export cookie 2573435527946852748 [10795.627122] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10795.639508] LustreError: 387612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10795.931261] Lustre: server umount lustre-MDT0001 complete [10805.681599] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:10:35 (1786176635) [10811.692148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10812.116387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10816.002661] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10817.513759] LustreError: 390132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.539147] LustreError: 390132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10822.431834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.825992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10826.097676] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10832.023838] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.099415] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10842.577981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.696557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10842.841400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10847.744794] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10855.097930] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10862.637618] Lustre: Permanently deactivating lustre-OST0000 [10862.654221] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10868.865217] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10871.590668] LustreError: 393115:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10871.599262] LustreError: 393115:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10871.604513] LustreError: 393115:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10874.467742] LustreError: 393163:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10874.510086] LustreError: 393163:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10882.630390] Lustre: server umount lustre-OST0000 complete [10885.757424] LustreError: 393413:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10885.786946] LustreError: 393413:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10889.185566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10889.198286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.214876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10889.218353] Lustre: Skipped 1 previous similar message [10892.256374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10892.257739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.376900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.384637] Lustre: Skipped 2 previous similar messages [10897.442132] Lustre: server umount lustre-MDT0000 complete [10900.850893] LustreError: 392820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176733 with bad export cookie 2573435527946853329 [10900.858467] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10900.859626] LustreError: 392820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10911.412592] Lustre: server umount lustre-MDT0001 complete [10919.192369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10923.756853] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10927.330484] LustreError: 394573:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10933.084782] Lustre: 394912:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.157@tcp [10934.780450] Lustre: server umount MGS complete [10945.383295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10945.679220] LustreError: 395637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.702988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10945.709358] Lustre: Skipped 1 previous similar message [10945.750235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10950.170166] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10951.136934] LustreError: 395638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.272776] LustreError: 395637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.898889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.079644] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10961.828847] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [10968.194781] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10973.992276] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10975.661760] Lustre: Permanently reactivating lustre-OST0000 [10975.699803] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10975.705107] Lustre: Skipped 2 previous similar messages [11004.897708] LustreError: 397466:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11012.127346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11012.457255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11012.473942] Lustre: Skipped 1 previous similar message [11018.068122] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11025.282739] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11029.986375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11029.991291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.999221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11035.700303] Lustre: server umount lustre-OST0000 complete [11040.230563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11040.258565] LustreError: Skipped 1 previous similar message [11040.265683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11040.274083] Lustre: Skipped 1 previous similar message [11045.572239] Lustre: server umount lustre-MDT0000 complete [11046.376115] LustreError: 395633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.390574] LustreError: 395633:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11049.281097] LustreError: 395618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786176882 with bad export cookie 2573435527946854379 [11049.287588] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.302062] LustreError: 395618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11049.591592] Lustre: server umount lustre-MDT0001 complete [11060.187640] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11062.680595] Key type lgssc unregistered [11063.068407] LNet: 400100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11063.078709] LNetError: 400100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11064.101892] LNet: Removed LNI 192.168.203.157@tcp [11065.039225] Key type .llcrypt unregistered [11065.040913] Key type ._llcrypt unregistered [11082.620960] Key type ._llcrypt registered [11082.627913] Key type .llcrypt registered [11082.849906] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [11097.199651] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11098.990572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11099.058989] alg: No test for adler32 (adler32-zlib) [11100.094275] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11100.385763] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11102.154068] Key type lgssc registered [11103.072756] Lustre: Echo OBD driver; http://www.lustre.org/ [11136.715411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11136.733272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11137.950919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11137.985940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11138.041861] Lustre: lustre-MDT0000: new disk, initializing [11138.096298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11138.109268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11142.485494] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11153.008407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.150299] Lustre: 403257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11153.214041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11153.218354] Lustre: Skipped 1 previous similar message [11153.288261] Lustre: lustre-MDT0001: new disk, initializing [11153.371498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11153.406288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11153.417704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11157.883318] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11168.288760] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11173.242081] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11178.755905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11179.051296] Lustre: lustre-OST0000: new disk, initializing [11179.056119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11179.062291] Lustre: 405227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11179.138836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11184.564739] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11185.687191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11185.700438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11185.756779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11194.217318] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11198.382403] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11198.595399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11202.563665] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11202.781479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11206.113875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11206.114792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11206.119179] LustreError: Skipped 1 previous similar message [11206.122418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11206.157979] Lustre: Skipped 1 previous similar message [11209.185083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11209.188642] Lustre: Skipped 2 previous similar messages [11210.555592] Lustre: server umount lustre-OST0000 complete [11214.304914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11214.312135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11214.317069] Lustre: Skipped 1 previous similar message [11214.330731] Lustre: Skipped 1 previous similar message [11219.430714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.453117] Lustre: Skipped 1 previous similar message [11220.167798] Lustre: server umount lustre-MDT0000 complete [11222.905072] LustreError: 403251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177056 with bad export cookie 13862692439299333763 [11222.917709] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11222.921880] LustreError: 403251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11223.171766] Lustre: server umount lustre-MDT0001 complete [11232.566786] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11234.499428] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:17:44 (1786177064) [11241.291036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.700051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11246.414736] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11247.075379] LustreError: 407688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11247.132885] LustreError: 407688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11251.171166] LustreError: 407687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11255.906679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11256.161798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11259.862975] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11267.406121] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11273.212066] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11279.788804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11279.993801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11287.702750] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11294.211142] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11296.303581] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11299.972800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11306.979138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.004197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11307.027763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11307.040762] LustreError: Skipped 1 previous similar message [11309.748496] Lustre: server umount lustre-OST0000 complete [11317.216682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.220195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.224815] Lustre: Skipped 2 previous similar messages [11317.231175] Lustre: Skipped 2 previous similar messages [11320.073428] Lustre: server umount lustre-MDT0000 complete [11322.337980] LustreError: 407688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11323.390332] LustreError: 407667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177156 with bad export cookie 13862692439299334638 [11323.397482] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11323.405434] LustreError: 407667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11323.656506] Lustre: server umount lustre-MDT0001 complete [11334.193219] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11336.669328] Key type lgssc unregistered [11336.860568] LNet: 411414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11336.872564] LNetError: 411414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11336.895804] LNet: Removed LNI 192.168.203.157@tcp [11337.700764] Key type .llcrypt unregistered [11337.702590] Key type ._llcrypt unregistered [11360.647287] Key type ._llcrypt registered [11360.649993] Key type .llcrypt registered [11360.734272] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11361.175429] Key type .llcrypt unregistered [11361.179077] Key type ._llcrypt unregistered [11378.879990] Key type ._llcrypt registered [11378.908899] Key type .llcrypt registered [11379.026287] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11379.700713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11379.742583] alg: No test for adler32 (adler32-zlib) [11381.092406] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11381.466304] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11383.254621] Key type lgssc registered [11384.847677] Lustre: Echo OBD driver; http://www.lustre.org/ [11391.953657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11391.967721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.275424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11397.279272] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11403.762580] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.818444] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11405.541643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11405.863098] LustreError: 413630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11405.903615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11410.913969] LustreError: 413626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11410.924658] LustreError: 413626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11411.316446] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11416.037642] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.057479] LustreError: 413631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11418.012372] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11418.100440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11423.215170] LustreError: 413625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11423.232802] LustreError: 413625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11425.338996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11425.683270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11430.525569] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11438.562818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11438.574992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11438.594963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11441.128424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11441.132557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11441.132583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11443.057690] Lustre: server umount lustre-OST0000 complete [11451.367933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11451.368787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11451.369665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11451.369671] Lustre: Skipped 1 previous similar message [11451.387316] Lustre: Skipped 1 previous similar message [11452.986994] Lustre: server umount lustre-MDT0000 complete [11456.415650] LustreError: 413611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177289 with bad export cookie 2602775111551288414 [11456.416686] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11456.425502] LustreError: 413611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11456.721148] Lustre: server umount lustre-MDT0001 complete [11466.750556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11469.218291] Key type lgssc unregistered [11469.460901] LNet: 416780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11469.477658] LNetError: 416780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11470.535946] LNet: Removed LNI 192.168.203.157@tcp [11471.317387] Key type .llcrypt unregistered [11471.324738] Key type ._llcrypt unregistered [11494.234813] Key type ._llcrypt registered [11494.237270] Key type .llcrypt registered [11494.416126] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11495.275062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11495.323138] alg: No test for adler32 (adler32-zlib) [11496.818188] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11497.084703] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11498.743149] Key type lgssc registered [11499.350897] Lustre: Echo OBD driver; http://www.lustre.org/ [11507.202625] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11507.216835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11522.719360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11527.839313] LustreError: 418169:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [11554.527232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11564.928473] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11572.490733] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11577.987962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11578.384720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11578.452369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11583.317499] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11583.464146] LustreError: 418194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.487517] LustreError: 418194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11588.579346] LustreError: 418195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.603150] LustreError: 418194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.609174] LustreError: 418194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11593.212261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.442549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11598.065723] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11605.043357] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11610.980627] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11618.696628] Lustre: server umount lustre-OST0000 complete [11624.433707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11624.441458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11624.446387] Lustre: Skipped 1 previous similar message [11624.447158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11624.457513] Lustre: Skipped 1 previous similar message [11629.542481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11629.559389] Lustre: Skipped 1 previous similar message [11633.692134] Lustre: server umount lustre-MDT0000 complete [11634.664749] LustreError: 419211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.618736] Lustre: 417506:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786177452/real 1786177452] req@00000000c5fb22d7 x1872942680010752/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786177468 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11635.694616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.541459] LustreError: 418175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177471 with bad export cookie 8225020234654037424 [11638.553805] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11639.161674] Lustre: server umount lustre-MDT0001 complete [11651.355447] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11653.425100] Key type lgssc unregistered [11653.617397] LNet: 421742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11653.628149] LNetError: 421742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11653.643095] LNet: Removed LNI 192.168.203.157@tcp [11654.593739] Key type .llcrypt unregistered [11654.598854] Key type ._llcrypt unregistered [11679.984806] Key type ._llcrypt registered [11679.986901] Key type .llcrypt registered [11680.097608] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11680.848467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11680.873139] alg: No test for adler32 (adler32-zlib) [11682.184083] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11682.428388] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11684.079149] Key type lgssc registered [11685.215752] Lustre: Echo OBD driver; http://www.lustre.org/ [11692.901310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11692.919654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.498156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11699.094766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11704.810626] LustreError: 423160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.847774] LustreError: 423160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11705.831671] LustreError: 423159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.441838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11706.696579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11711.408902] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11718.477218] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11724.397582] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11732.760437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11733.043844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11739.266386] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11740.274682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11746.989552] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11752.928245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11752.932717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11752.958418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11754.845780] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [11757.029806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11757.032200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11758.152127] Lustre: server umount lustre-MDT0000 complete [11759.968307] LustreError: 423156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11760.643131] LustreError: 423141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.57@tcp arrived at 1786177593 with bad export cookie 14062021544315970034 [11760.650321] LustreError: 423141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11762.257278] LustreError: 425690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.271302] LustreError: 425690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11763.161697] LustreError: 423140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786177596 with bad export cookie 14062021544315969145 [11763.194550] LustreError: 423140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11763.213037] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.268519] LustreError: 423159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.269175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11767.275269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11767.300670] LustreError: 423159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11767.309047] Lustre: Skipped 1 previous similar message [11767.314977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11767.314986] Lustre: Skipped 2 previous similar messages [11769.918168] Lustre: server umount lustre-OST0000 complete [11777.464945] Lustre: server umount lustre-MDT0001 complete [11789.084406] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11791.445644] Key type lgssc unregistered [11791.653743] LNet: 426849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11791.658766] LNetError: 426849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11792.694534] LNet: Removed LNI 192.168.203.157@tcp [11793.244795] Key type .llcrypt unregistered [11793.246859] Key type ._llcrypt unregistered [11817.499540] Key type ._llcrypt registered [11817.502367] Key type .llcrypt registered [11817.670275] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11818.830581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11819.036225] alg: No test for adler32 (adler32-zlib) [11820.515681] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11820.699188] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11822.369158] Key type lgssc registered [11823.277738] Lustre: Echo OBD driver; http://www.lustre.org/ [11830.739445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11830.749689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11832.193841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11836.085640] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11842.529225] LustreError: 428269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.540520] LustreError: 428269:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11843.420642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11843.553793] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11843.664527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11848.072924] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11855.861602] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11860.519586] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11866.687844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11867.108762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11872.729765] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [11878.451294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11881.496171] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11888.610398] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11888.646254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11888.653728] Lustre: Skipped 1 previous similar message [11890.010623] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [11893.029771] Lustre: server umount lustre-MDT0001 complete [11893.738610] LustreError: 430681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.781314] LustreError: 430681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11898.859889] LustreError: 428265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11898.871907] LustreError: 428265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11903.973536] LustreError: 428268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.978710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11903.977091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11904.050969] Lustre: Skipped 1 previous similar message [11904.062171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11905.088700] Lustre: server umount lustre-OST0000 complete [11910.275949] Lustre: server umount lustre-MDT0000 complete [11923.124994] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [11925.665384] Key type lgssc unregistered [11925.896661] LNet: 431955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11925.902230] LNetError: 431955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11925.912241] LNet: Removed LNI 192.168.203.157@tcp [11926.563439] Key type .llcrypt unregistered [11926.565755] Key type ._llcrypt unregistered [11951.949618] Key type ._llcrypt registered [11951.952805] Key type .llcrypt registered [11952.029933] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [11966.551525] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [11968.700109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11968.821043] alg: No test for adler32 (adler32-zlib) [11969.810645] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [11969.997087] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [11971.681111] Key type lgssc registered [11972.900558] Lustre: Echo OBD driver; http://www.lustre.org/ [12005.793201] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [12019.892314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12019.923684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.302678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12021.325669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12021.397578] Lustre: lustre-MDT0000: new disk, initializing [12021.457450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12021.474660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12026.295379] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12041.342621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.563594] Lustre: lustre-OST0000: new disk, initializing [12041.566853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12041.571065] Lustre: Skipped 1 previous similar message [12041.575725] Lustre: 436990:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12041.632587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12047.419363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12047.424555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12047.436485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12048.385679] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12057.472889] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12072.307782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.471640] Lustre: 435972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12072.539299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12072.594808] Lustre: lustre-MDT0001: new disk, initializing [12072.762763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12072.945116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12072.993718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12077.057898] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12087.682126] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12087.847625] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12092.266348] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12092.459564] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12099.042920] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [12102.631780] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12102.642020] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12102.644352] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12102.648557] Lustre: Skipped 1 previous similar message [12103.998303] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.57@tcp (stopping) [12107.759116] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12107.774238] Lustre: Skipped 1 previous similar message [12112.360476] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12112.499153] Lustre: server umount lustre-MDT0001 complete [12117.986216] LustreError: 437516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.002200] LustreError: 437516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12123.109020] LustreError: 435978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12123.139957] LustreError: 435978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12128.246324] LustreError: 437516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.256827] LustreError: 437516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12129.249308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12129.254868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12129.261459] Lustre: Skipped 1 previous similar message [12129.264528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12129.267645] Lustre: Skipped 1 previous similar message [12133.351948] LustreError: 435983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.744486] Lustre: server umount lustre-OST0000 complete [12138.466666] LustreError: 435984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12145.903738] Lustre: server umount lustre-MDT0000 complete [12159.693425] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [12162.361456] Key type lgssc unregistered [12162.713857] LNet: 440744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12162.718734] LNetError: 440744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12162.730185] LNet: Removed LNI 192.168.203.157@tcp [12163.443444] Key type .llcrypt unregistered [12163.445364] Key type ._llcrypt unregistered [12205.435419] Key type ._llcrypt registered [12205.441295] Key type .llcrypt registered [12205.556263] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [12205.862574] Key type .llcrypt unregistered [12205.869544] Key type ._llcrypt unregistered [12221.298904] Key type ._llcrypt registered [12221.300941] Key type .llcrypt registered [12221.431708] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [12223.074791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12223.209362] alg: No test for adler32 (adler32-zlib) [12224.619252] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [12224.913549] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [12226.647128] Key type lgssc registered [12227.584488] Lustre: Echo OBD driver; http://www.lustre.org/ [12254.430601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12254.486754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.745969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12255.770317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12256.005116] Lustre: lustre-MDT0000: new disk, initializing [12256.066496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.084206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12260.257547] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12269.848053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.951580] Lustre: 444125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12269.992689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12269.996548] Lustre: Skipped 1 previous similar message [12270.044845] Lustre: lustre-MDT0001: new disk, initializing [12270.117938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12270.152482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12270.163426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12274.013919] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12282.702406] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12288.472442] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12294.958442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12295.128916] Lustre: lustre-OST0000: new disk, initializing [12295.132349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12295.142873] Lustre: 446092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12295.214218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12301.418843] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12304.964367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12304.975319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12305.173407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12313.200215] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12320.750868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.766165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.775739] Lustre: Skipped 2 previous similar messages [12320.779130] Lustre: Skipped 1 previous similar message [12325.332991] Lustre: server umount lustre-MDT0000 complete [12329.821873] LustreError: 444919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178162 with bad export cookie 8383590773848155443 [12329.823029] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.856877] LustreError: 444919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12330.329939] Lustre: server umount lustre-MDT0001 complete [12344.776788] Lustre: server umount lustre-OST0000 complete [12355.610365] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:36:25 (1786178185) [12362.435703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.793754] LustreError: 448120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12362.814956] LustreError: 448120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12362.884690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12367.657208] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12368.353676] LustreError: 448121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12372.459668] LustreError: 448120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12375.069504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.276823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12380.685319] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12387.459472] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12392.631465] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12412.266821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12412.401427] Lustre: Found index 0 for lustre-OST0000, updating log [12412.536331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12417.836334] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [12425.762766] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12433.004700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12443.643116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12443.649590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.659125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12443.663121] Lustre: Skipped 1 previous similar message [12444.710488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12444.728523] Lustre: Skipped 1 previous similar message [12449.300955] Lustre: server umount lustre-OST0000 complete [12466.147913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.148900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12466.164674] Lustre: Skipped 2 previous similar messages [12466.176096] Lustre: Skipped 1 previous similar message [12470.925507] Lustre: server umount lustre-MDT0000 complete [12471.269959] LustreError: 448120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12471.293411] LustreError: 448120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12474.901338] LustreError: 449962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786178308 with bad export cookie 8383590773848156836 [12474.907598] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12474.918059] LustreError: 449962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12475.318863] Lustre: server umount lustre-MDT0001 complete [12475.812940] LNet: 1 local NIs in recovery (showing 1): 192.168.203.157@tcp [12484.898778] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:38:34 (1786178314) [13104.826492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13105.150379] LustreError: 454558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13105.166426] LustreError: 454558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13105.215460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13109.138718] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13110.250192] LustreError: 454559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13115.393867] LustreError: 454558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13118.257372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13118.586322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13123.495635] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13130.013119] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13134.625274] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13153.704272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13153.952955] Lustre: Found index 0 for lustre-OST0000, updating log [13154.339168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13160.205626] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13165.216593] LNet: 1 local NIs in recovery (showing 1): 192.168.203.157@tcp [13167.508731] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13169.745430] Lustre: server umount lustre-OST0000 complete [13174.757331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13174.766854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13174.782813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13175.845187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13175.849912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.864548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13176.872154] Lustre: Skipped 2 previous similar messages [13179.000911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13179.006654] Lustre: Skipped 3 previous similar messages [13180.241243] Lustre: server umount lustre-MDT0000 complete [13181.156262] LustreError: 454559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13181.179583] LustreError: 454559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13184.489561] LustreError: 454538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179017 with bad export cookie 8383590773848157914 [13184.512100] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13184.520161] LustreError: 454538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13185.036549] Lustre: server umount lustre-MDT0001 complete [13217.492904] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [13226.495172] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [13264.883972] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:51:34 (1786179094) [13281.413883] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [13283.737234] Key type lgssc unregistered [13284.260888] LNet: 461176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13284.265366] LNetError: 461176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13407.653938] LNet: Removed LNI 192.168.203.157@tcp [13408.289167] Key type .llcrypt unregistered [13408.290759] Key type ._llcrypt unregistered [13424.809534] Key type ._llcrypt registered [13424.812630] Key type .llcrypt registered [13424.917066] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [13425.893646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13425.976368] alg: No test for adler32 (adler32-zlib) [13427.569539] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [13427.839653] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [13429.495167] Key type lgssc registered [13430.458360] Lustre: Echo OBD driver; http://www.lustre.org/ [13434.737344] LNet: Added LNI 192.168.203.157@tcp100 [8/256/0/180] [13435.501292] LNet: Added LNI 192.168.203.157@tcp101 [8/256/0/180] [13436.156839] LNet: Added LNI 192.168.203.157@tcp102 [8/256/0/180] [13437.065634] LNet: Added LNI 192.168.203.157@tcp103 [8/256/0/180] [13437.942408] LNet: Added LNI 192.168.203.157@tcp104 [8/256/0/180] [13438.823123] LNet: Added LNI 192.168.203.157@tcp105 [8/256/0/180] [13439.449629] LNet: Added LNI 192.168.203.157@tcp106 [8/256/0/180] [13440.451253] LNet: Added LNI 192.168.203.157@tcp107 [8/256/0/180] [13441.443208] LNet: Added LNI 192.168.203.157@tcp108 [8/256/0/180] [13442.625213] LNet: Added LNI 192.168.203.157@tcp109 [8/256/0/180] [13443.354681] LNet: Added LNI 192.168.203.157@tcp110 [8/256/0/180] [13444.089301] LNet: Added LNI 192.168.203.157@tcp111 [8/256/0/180] [13445.037635] LNet: Added LNI 192.168.203.157@tcp112 [8/256/0/180] [13446.200577] LNet: Added LNI 192.168.203.157@tcp113 [8/256/0/180] [13446.940852] LNet: Added LNI 192.168.203.157@tcp114 [8/256/0/180] [13447.979984] LNet: Added LNI 192.168.203.157@tcp115 [8/256/0/180] [13448.931395] LNet: Added LNI 192.168.203.157@tcp116 [8/256/0/180] [13449.980430] LNet: Added LNI 192.168.203.157@tcp117 [8/256/0/180] [13450.657049] LNet: Added LNI 192.168.203.157@tcp118 [8/256/0/180] [13451.703389] LNet: Added LNI 192.168.203.157@tcp119 [8/256/0/180] [13452.472800] LNet: Added LNI 192.168.203.157@tcp120 [8/256/0/180] [13453.461569] LNet: Added LNI 192.168.203.157@tcp121 [8/256/0/180] [13454.391288] LNet: Added LNI 192.168.203.157@tcp122 [8/256/0/180] [13455.295930] LNet: Added LNI 192.168.203.157@tcp123 [8/256/0/180] [13456.263604] LNet: Added LNI 192.168.203.157@tcp124 [8/256/0/180] [13457.084244] LNet: Added LNI 192.168.203.157@tcp125 [8/256/0/180] [13457.788775] LNet: Added LNI 192.168.203.157@tcp126 [8/256/0/180] [13458.592337] LNet: Added LNI 192.168.203.157@tcp127 [8/256/0/180] [13459.272419] LNet: Added LNI 192.168.203.157@tcp128 [8/256/0/180] [13459.932558] LNet: Added LNI 192.168.203.157@tcp129 [8/256/0/180] [13460.540017] LNet: Added LNI 192.168.203.157@tcp130 [8/256/0/180] [13461.155070] LNet: Added LNI 192.168.203.157@tcp131 [8/256/0/180] [13461.789304] LNet: Added LNI 192.168.203.157@tcp132 [8/256/0/180] [13462.602993] LNet: Added LNI 192.168.203.157@tcp133 [8/256/0/180] [13463.220485] LNet: Added LNI 192.168.203.157@tcp134 [8/256/0/180] [13464.139906] LNet: Added LNI 192.168.203.157@tcp135 [8/256/0/180] [13470.398078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13470.416118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13471.595168] Lustre: 464209:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.157@tcp: can't add NID 192.168.203.157@tcp130: rc = -75 [13471.743229] LNetError: 461666:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13471.756677] LNetError: Protocol error connecting to 192.168.203.157@tcp121 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13471.762914] LNetError: Skipped 1 previous similar message [13471.856979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13471.878655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13471.954371] Lustre: lustre-MDT0000: new disk, initializing [13472.047759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13472.070063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13472.351218] LNet: 4 local NIs in recovery (showing 4): 192.168.203.157@tcp125, 192.168.203.157@tcp126, 192.168.203.157@tcp121, 192.168.203.157@tcp128 [13472.354129] LNetError: 461667:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13472.396540] LNetError: 461667:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13472.424702] LNetError: Protocol error connecting to 192.168.203.157@tcp102 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13472.434951] LNetError: Skipped 5 previous similar messages [13476.395396] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13487.924614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.049899] Lustre: 464226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13488.094485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13488.098352] Lustre: Skipped 1 previous similar message [13488.152945] Lustre: lustre-MDT0001: new disk, initializing [13488.187967] Lustre: 465027:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.157@tcp: can't add NID 192.168.203.157@tcp130: rc = -75 [13488.197478] Lustre: 465027:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13488.319274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13488.379388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13488.391596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13492.765200] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13497.420856] Lustre: 465442:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.157@tcp: can't add NID 192.168.203.157@tcp130: rc = -75 [13497.453128] Lustre: 465442:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13501.024971] LNetError: 464232:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13501.037310] LNetError: Protocol error connecting to 192.168.203.157@tcp118 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13501.878590] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13504.096892] LNetError: 464240:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13504.109980] LNetError: Protocol error connecting to 192.168.203.157@tcp124 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13506.535278] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13510.370910] LNetError: 464230:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13510.490776] LNetError: 464230:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [13510.497667] LNetError: Protocol error connecting to 192.168.203.157@tcp120 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13510.509209] LNetError: Skipped 1 previous similar message [13513.130250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13513.373748] Lustre: lustre-OST0000: new disk, initializing [13513.377710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13513.383978] Lustre: 466199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13513.470455] Lustre: 466204:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.157@tcp: can't add NID 192.168.203.157@tcp130: rc = -75 [13513.481232] Lustre: 466204:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13513.530813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13519.443947] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13519.503962] Lustre: 466554:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.157@tcp: can't add NID 192.168.203.157@tcp130: rc = -75 [13519.518239] Lustre: 466554:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13519.567170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13519.578361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13519.830958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13520.617966] LNetError: 461667:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13520.688875] LNetError: 461667:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13520.724351] LNetError: Protocol error connecting to 192.168.203.157@tcp122 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13520.759095] LNetError: Skipped 4 previous similar messages [13529.119377] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13537.062845] LNetError: 464229:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.157 [13537.079578] LNetError: 464229:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13537.090741] LNetError: Protocol error connecting to 192.168.203.157@tcp102 at host 192.168.203.157:988: is it running a compatible version of Lustre? [13537.100027] LNetError: Skipped 7 previous similar messages [13540.321527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13540.321810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.328628] LustreError: Skipped 1 previous similar message [13540.330245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13540.370149] Lustre: Skipped 1 previous similar message [13542.387490] Lustre: server umount lustre-OST0000 complete [13550.568849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13550.571223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13550.571789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13550.571793] Lustre: Skipped 1 previous similar message [13550.580486] Lustre: Skipped 1 previous similar message [13551.796993] Lustre: server umount lustre-MDT0000 complete [13555.502206] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179388 with bad export cookie 15084463893668445623 [13555.504023] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13555.514240] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13556.016907] Lustre: server umount lustre-MDT0001 complete [13566.582472] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [13569.374873] Key type lgssc unregistered [13569.819975] LNet: 468116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13569.827614] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13570.847171] LNet: Removed LNI 192.168.203.157@tcp [13570.851289] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13570.858847] LNet: Removed LNI 192.168.203.157@tcp100 [13571.871319] LNet: Removed LNI 192.168.203.157@tcp101 [13571.879643] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13571.898416] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13571.912340] LNet: Removed LNI 192.168.203.157@tcp102 [13572.959612] LNet: Removed LNI 192.168.203.157@tcp103 [13573.983156] LNet: Removed LNI 192.168.203.157@tcp104 [13573.985529] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13574.014675] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13574.020546] LNet: Removed LNI 192.168.203.157@tcp105 [13575.071755] LNet: Removed LNI 192.168.203.157@tcp106 [13576.095161] LNet: Removed LNI 192.168.203.157@tcp107 [13577.119504] LNet: Removed LNI 192.168.203.157@tcp108 [13578.144179] LNet: Removed LNI 192.168.203.157@tcp109 [13578.146483] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13578.172488] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13578.181693] LNet: Removed LNI 192.168.203.157@tcp110 [13579.232696] LNet: Removed LNI 192.168.203.157@tcp111 [13580.257405] LNet: Removed LNI 192.168.203.157@tcp112 [13581.281145] LNet: Removed LNI 192.168.203.157@tcp113 [13582.303137] LNet: Removed LNI 192.168.203.157@tcp114 [13583.327168] LNet: Removed LNI 192.168.203.157@tcp115 [13584.351161] LNet: Removed LNI 192.168.203.157@tcp116 [13585.375153] LNet: Removed LNI 192.168.203.157@tcp117 [13586.399480] LNet: Removed LNI 192.168.203.157@tcp118 [13586.406594] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13586.423735] LNetError: 468116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13586.440439] LNet: Removed LNI 192.168.203.157@tcp119 [13587.487146] LNet: Removed LNI 192.168.203.157@tcp120 [13588.511202] LNet: Removed LNI 192.168.203.157@tcp121 [13589.552598] LNet: Removed LNI 192.168.203.157@tcp122 [13590.559670] LNet: Removed LNI 192.168.203.157@tcp123 [13591.583375] LNet: Removed LNI 192.168.203.157@tcp124 [13592.607297] LNet: Removed LNI 192.168.203.157@tcp125 [13593.631280] LNet: Removed LNI 192.168.203.157@tcp126 [13594.658730] LNet: Removed LNI 192.168.203.157@tcp127 [13595.681215] LNet: Removed LNI 192.168.203.157@tcp128 [13596.707514] LNet: Removed LNI 192.168.203.157@tcp129 [13596.709799] LNet: Removed LNI 192.168.203.157@tcp130 [13596.712917] LNet: Removed LNI 192.168.203.157@tcp131 [13596.715895] LNet: Removed LNI 192.168.203.157@tcp132 [13596.717973] LNet: Removed LNI 192.168.203.157@tcp133 [13596.724076] LNet: Removed LNI 192.168.203.157@tcp134 [13596.737917] LNet: Removed LNI 192.168.203.157@tcp135 [13597.664834] Key type .llcrypt unregistered [13597.666914] Key type ._llcrypt unregistered [13620.174656] Key type ._llcrypt registered [13620.177254] Key type .llcrypt registered [13620.330646] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [13634.989951] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [13637.383311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13637.437892] alg: No test for adler32 (adler32-zlib) [13638.604536] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [13638.777572] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [13640.407162] Key type lgssc registered [13641.709392] Lustre: Echo OBD driver; http://www.lustre.org/ [13676.819824] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:58:26 (1786179506) [13695.607563] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [13697.516631] Key type lgssc unregistered [13697.861771] LNet: 472208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13697.866667] LNetError: 472208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13697.879900] LNet: Removed LNI 192.168.203.157@tcp [13699.151125] Key type .llcrypt unregistered [13699.160892] Key type ._llcrypt unregistered [13716.511215] Key type ._llcrypt registered [13716.512878] Key type .llcrypt registered [13716.762883] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [13720.295272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13720.486366] alg: No test for adler32 (adler32-zlib) [13721.998907] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [13722.501101] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [13724.151235] Key type lgssc registered [13725.788833] Lustre: Echo OBD driver; http://www.lustre.org/ [13734.817100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13734.915455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13736.139941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13736.173155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13736.223345] Lustre: lustre-MDT0000: new disk, initializing [13736.277265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13736.295892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13740.529741] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13751.085328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13751.210181] Lustre: 473420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13751.279899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13751.284261] Lustre: Skipped 1 previous similar message [13751.338553] Lustre: lustre-MDT0001: new disk, initializing [13751.433876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13751.460629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13751.473052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13756.449749] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13766.290578] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13772.794596] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13778.784300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13778.982491] Lustre: lustre-OST0000: new disk, initializing [13778.985770] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13778.990382] Lustre: 475391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13779.087100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13785.501264] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [13788.330162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13788.348487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13788.586439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13794.223057] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13797.691919] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing lnet_if_list [13802.487954] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13802.508823] LNet: Added LNI 192.168.203.157@tcp10 [8/256/0/180] [13808.026765] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13808.056438] LNet: Added LNI 192.168.203.157@tcp11 [8/256/0/180] [13811.781113] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13811.805617] LNet: Added LNI 192.168.203.157@tcp12 [8/256/0/180] [13815.797639] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13815.817514] LNet: Added LNI 192.168.203.157@tcp13 [8/256/0/180] [13820.635502] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13820.657372] LNet: Added LNI 192.168.203.157@tcp14 [8/256/0/180] [13825.205885] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13825.225781] LNet: Added LNI 192.168.203.157@tcp15 [8/256/0/180] [13829.926959] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13829.958632] LNet: Added LNI 192.168.203.157@tcp16 [8/256/0/180] [13835.228248] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13835.251038] LNet: Added LNI 192.168.203.157@tcp17 [8/256/0/180] [13840.005556] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13840.026868] LNet: Added LNI 192.168.203.157@tcp18 [8/256/0/180] [13844.521560] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13844.546707] LNet: Added LNI 192.168.203.157@tcp19 [8/256/0/180] [13849.331379] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13849.352710] LNet: Added LNI 192.168.203.157@tcp20 [8/256/0/180] [13855.608426] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13855.629961] LNet: Added LNI 192.168.203.157@tcp21 [8/256/0/180] [13860.552566] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13860.573718] LNet: Added LNI 192.168.203.157@tcp22 [8/256/0/180] [13866.218042] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13866.238832] LNet: Added LNI 192.168.203.157@tcp23 [8/256/0/180] [13872.283566] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13872.322968] LNet: Added LNI 192.168.203.157@tcp24 [8/256/0/180] [13879.136498] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13879.175734] LNet: Added LNI 192.168.203.157@tcp25 [8/256/0/180] [13885.475475] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13885.497194] LNet: Added LNI 192.168.203.157@tcp26 [8/256/0/180] [13892.133391] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13892.171601] LNet: Added LNI 192.168.203.157@tcp27 [8/256/0/180] [13898.482612] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13898.520561] LNet: Added LNI 192.168.203.157@tcp28 [8/256/0/180] [13904.375977] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13904.403215] LNet: Added LNI 192.168.203.157@tcp29 [8/256/0/180] [13910.544481] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13910.568955] LNet: Added LNI 192.168.203.157@tcp30 [8/256/0/180] [13917.117901] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13917.136783] LNet: Added LNI 192.168.203.157@tcp31 [8/256/0/180] [13923.289634] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13923.319147] LNet: Added LNI 192.168.203.157@tcp32 [8/256/0/180] [13930.309671] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13930.343947] LNet: Added LNI 192.168.203.157@tcp33 [8/256/0/180] [13937.240881] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13937.266784] LNet: Added LNI 192.168.203.157@tcp34 [8/256/0/180] [13944.544978] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13944.569617] LNet: Added LNI 192.168.203.157@tcp35 [8/256/0/180] [13953.236642] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13953.263152] LNet: Added LNI 192.168.203.157@tcp36 [8/256/0/180] [13961.360479] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13961.380430] LNet: Added LNI 192.168.203.157@tcp37 [8/256/0/180] [13968.881376] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13968.901658] LNet: Added LNI 192.168.203.157@tcp38 [8/256/0/180] [13977.325491] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13977.349851] LNet: Added LNI 192.168.203.157@tcp39 [8/256/0/180] [13985.192418] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13985.212684] LNet: Added LNI 192.168.203.157@tcp40 [8/256/0/180] [13992.592588] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13992.624305] LNet: Added LNI 192.168.203.157@tcp41 [8/256/0/180] [14001.051365] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14001.148242] LNet: Added LNI 192.168.203.157@tcp42 [8/256/0/180] [14009.639930] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14009.672867] LNet: Added LNI 192.168.203.157@tcp43 [8/256/0/180] [14017.711367] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14017.750506] LNet: Added LNI 192.168.203.157@tcp44 [8/256/0/180] [14026.476247] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14026.499038] LNet: Added LNI 192.168.203.157@tcp45 [8/256/0/180] [14034.264834] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14034.285276] LNet: Added LNI 192.168.203.157@tcp46 [8/256/0/180] [14043.005730] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14043.027849] LNet: Added LNI 192.168.203.157@tcp47 [8/256/0/180] [14051.205665] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14051.226587] LNet: Added LNI 192.168.203.157@tcp48 [8/256/0/180] [14059.692462] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14059.714575] LNet: Added LNI 192.168.203.157@tcp49 [8/256/0/180] [14067.525372] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14067.564903] LNet: Added LNI 192.168.203.157@tcp50 [8/256/0/180] [14088.673741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14088.677670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14088.688884] Lustre: Skipped 2 previous similar messages [14088.693665] Lustre: Skipped 1 previous similar message [14090.515247] Lustre: server umount lustre-MDT0000 complete [14093.468242] LustreError: 473413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786179926 with bad export cookie 856433818757997495 [14093.475092] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14093.482329] LustreError: 473413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14093.808231] Lustre: server umount lustre-MDT0001 complete [14109.151145] Lustre: 472692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786179926/real 1786179926] req@00000000cc68acf3 x1872945013238784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786179942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14109.174073] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14110.928709] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14115.322124] Lustre: 472694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786179932/real 1786179932] req@0000000006a2c3a8 x1872945013238912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786179948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14118.965425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14119.391531] Lustre: 472695:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786179936/real 1786179936] req@000000008077ccfb x1872945013239168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786179952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14124.535435] Lustre: 472693:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786179941/real 1786179941] req@0000000075a923de x1872945013239296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786179957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14138.016927] LustreError: 472691:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000000dac8a8 x1872945013240192/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14138.627396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14143.021417] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14143.970128] LustreError: 475385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14143.991882] LustreError: 475385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14149.091731] LustreError: 475386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14152.246269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14152.567891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14153.606898] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14154.669978] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14156.890130] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14163.630745] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14169.362188] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14188.512584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14188.516916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14188.525053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14188.528392] Lustre: Skipped 1 previous similar message [14190.049943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14190.064567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14192.479763] Lustre: server umount lustre-OST0000 complete [14195.681171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14195.681814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14195.696245] Lustre: Skipped 2 previous similar messages [14195.702390] Lustre: Skipped 1 previous similar message [14200.802500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14200.817617] Lustre: Skipped 2 previous similar messages [14201.849594] Lustre: server umount lustre-MDT0000 complete [14205.093941] LustreError: 474216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180038 with bad export cookie 856433818758000057 [14205.101756] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14205.106787] LustreError: 474216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14205.468565] Lustre: server umount lustre-MDT0001 complete [14215.122857] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [14217.472156] Key type lgssc unregistered [14217.862666] LNet: 502425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14217.869338] LNetError: 502425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14217.873481] LNet: Removed LNI 192.168.203.157@tcp [14217.875533] LNet: Removed LNI 192.168.203.157@tcp10 [14217.878187] LNet: Removed LNI 192.168.203.157@tcp11 [14217.880751] LNet: Removed LNI 192.168.203.157@tcp12 [14217.883439] LNet: Removed LNI 192.168.203.157@tcp13 [14217.886036] LNet: Removed LNI 192.168.203.157@tcp14 [14217.888620] LNet: Removed LNI 192.168.203.157@tcp15 [14217.892140] LNet: Removed LNI 192.168.203.157@tcp16 [14217.895251] LNet: Removed LNI 192.168.203.157@tcp17 [14217.897912] LNet: Removed LNI 192.168.203.157@tcp18 [14217.900846] LNet: Removed LNI 192.168.203.157@tcp19 [14217.903666] LNet: Removed LNI 192.168.203.157@tcp20 [14217.906248] LNet: Removed LNI 192.168.203.157@tcp21 [14217.910749] LNet: Removed LNI 192.168.203.157@tcp22 [14217.913420] LNet: Removed LNI 192.168.203.157@tcp23 [14217.917326] LNet: Removed LNI 192.168.203.157@tcp24 [14217.921433] LNet: Removed LNI 192.168.203.157@tcp25 [14217.925748] LNet: Removed LNI 192.168.203.157@tcp26 [14217.930271] LNet: Removed LNI 192.168.203.157@tcp27 [14217.935315] LNet: Removed LNI 192.168.203.157@tcp28 [14217.940360] LNet: Removed LNI 192.168.203.157@tcp29 [14217.944573] LNet: Removed LNI 192.168.203.157@tcp30 [14217.947592] LNet: Removed LNI 192.168.203.157@tcp31 [14217.951539] LNet: Removed LNI 192.168.203.157@tcp32 [14217.957469] LNet: Removed LNI 192.168.203.157@tcp33 [14217.963532] LNet: Removed LNI 192.168.203.157@tcp34 [14217.969391] LNet: Removed LNI 192.168.203.157@tcp35 [14217.971704] LNet: Removed LNI 192.168.203.157@tcp36 [14217.980374] LNet: Removed LNI 192.168.203.157@tcp37 [14217.982938] LNet: Removed LNI 192.168.203.157@tcp38 [14217.985356] LNet: Removed LNI 192.168.203.157@tcp39 [14217.987136] LNet: Removed LNI 192.168.203.157@tcp40 [14217.990159] LNet: Removed LNI 192.168.203.157@tcp41 [14217.993432] LNet: Removed LNI 192.168.203.157@tcp42 [14217.997815] LNet: Removed LNI 192.168.203.157@tcp43 [14218.000400] LNet: Removed LNI 192.168.203.157@tcp44 [14218.002599] LNet: Removed LNI 192.168.203.157@tcp45 [14218.005961] LNet: Removed LNI 192.168.203.157@tcp46 [14218.008809] LNet: Removed LNI 192.168.203.157@tcp47 [14218.011233] LNet: Removed LNI 192.168.203.157@tcp48 [14218.013357] LNet: Removed LNI 192.168.203.157@tcp49 [14218.022474] LNet: Removed LNI 192.168.203.157@tcp50 [14218.751016] Key type .llcrypt unregistered [14218.760435] Key type ._llcrypt unregistered [14236.152888] Key type ._llcrypt registered [14236.154623] Key type .llcrypt registered [14236.236712] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [14248.372154] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14249.051028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14249.113816] alg: No test for adler32 (adler32-zlib) [14250.244293] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [14250.469655] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [14252.135170] Key type lgssc registered [14253.162731] Lustre: Echo OBD driver; http://www.lustre.org/ [14285.373392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14285.444874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14286.667688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14286.727551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14286.863771] Lustre: lustre-MDT0000: new disk, initializing [14287.050649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14287.067927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14291.368602] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14301.006556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14301.108265] Lustre: 505598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14301.149740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14301.153328] Lustre: Skipped 1 previous similar message [14301.209938] Lustre: lustre-MDT0001: new disk, initializing [14301.282694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14301.311630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14301.326597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14305.718159] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14313.456435] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14319.805689] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14326.103850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14326.299422] Lustre: *** cfs_fail_loc=90f, val=0*** [14326.346248] Lustre: lustre-OST0000: new disk, initializing [14326.349230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14326.354186] Lustre: 507614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14326.421722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14331.662074] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14332.987951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14333.004991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14333.161449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14339.954168] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14348.775863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14348.779406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14348.798210] LustreError: Skipped 1 previous similar message [14348.800946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14348.809102] Lustre: Skipped 1 previous similar message [14353.893166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14353.904176] Lustre: Skipped 2 previous similar messages [14354.605315] Lustre: server umount lustre-OST0000 complete [14359.014968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14359.018027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14359.027890] Lustre: Skipped 1 previous similar message [14359.057605] Lustre: Skipped 1 previous similar message [14364.135813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14364.144326] Lustre: Skipped 1 previous similar message [14364.739393] Lustre: server umount lustre-MDT0000 complete [14368.478461] LustreError: 506395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180201 with bad export cookie 5250675980050350428 [14368.486131] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14368.502285] LustreError: 506395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14368.898994] Lustre: server umount lustre-MDT0001 complete [14379.072745] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [14381.735786] Key type lgssc unregistered [14381.986583] LNet: 509524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14382.002203] LNetError: 509524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14382.063560] LNet: Removed LNI 192.168.203.157@tcp [14382.776438] Key type .llcrypt unregistered [14382.778698] Key type ._llcrypt unregistered [14399.589575] Key type ._llcrypt registered [14399.591787] Key type .llcrypt registered [14399.674308] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14400.796372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14400.888226] alg: No test for adler32 (adler32-zlib) [14401.888802] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [14402.094284] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [14403.832444] Key type lgssc registered [14404.673717] Lustre: Echo OBD driver; http://www.lustre.org/ [14412.041324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14412.056561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14413.346697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14417.650285] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14423.520100] LustreError: 510774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14423.530963] LustreError: 510774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14424.555627] LustreError: 510775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14425.605657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14425.928091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14430.084632] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14437.078876] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14442.510801] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14448.943534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14449.181614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14454.595684] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14460.593862] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14470.340243] loop0: detected capacity change from 0 to 204800000 [14470.490841] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14470.499874] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14470.514970] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14475.939603] loop0: detected capacity change from 0 to 204800000 [14476.226149] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14476.234344] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14476.250727] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14480.605319] loop0: detected capacity change from 0 to 204800000 [14485.846306] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14485.978459] Lustre: fs15246-OST0000: new disk, initializing [14485.983312] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14485.991098] Lustre: 514368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14486.044365] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14491.423862] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14501.701416] loop1: detected capacity change from 0 to 204800000 [14506.963295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14507.055871] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14507.097748] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14507.153524] Lustre: fs15246-MDT0000: new disk, initializing [14507.275283] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14507.301704] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14507.404684] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14508.453466] Lustre: 515489:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a9ded31d x1872945726141056/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786180357 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14511.376386] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14513.509996] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14513.515744] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14513.680631] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14533.097807] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14533.127257] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14536.855546] Lustre: server umount fs15246-MDT0000 complete [14538.225619] LustreError: 512690:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14543.330690] LustreError: 512690:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14545.916547] Lustre: server umount fs15246-OST0000 complete [14557.152932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14557.157432] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14557.178353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14557.666877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14558.696294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14558.711159] Lustre: Skipped 2 previous similar messages [14560.964457] Lustre: server umount lustre-OST0000 complete [14568.932663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14568.944078] Lustre: Skipped 1 previous similar message [14568.949127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.283610] Lustre: server umount lustre-MDT0000 complete [14574.051311] LustreError: 510775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14574.071746] LustreError: 510775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14574.662401] LustreError: 512617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180407 with bad export cookie 16091259525003398930 [14574.664898] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14574.680281] LustreError: 512617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14575.011740] Lustre: server umount lustre-MDT0001 complete [14585.638839] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [14588.327535] Key type lgssc unregistered [14588.614795] LNet: 518586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14588.622935] LNetError: 518586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14588.634887] LNet: Removed LNI 192.168.203.157@tcp [14589.296379] Key type .llcrypt unregistered [14589.299076] Key type ._llcrypt unregistered [14605.775416] Key type ._llcrypt registered [14605.781478] Key type .llcrypt registered [14605.882869] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14607.076691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14607.102831] alg: No test for adler32 (adler32-zlib) [14608.098295] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [14608.286743] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [14609.951923] Key type lgssc registered [14610.775052] Lustre: Echo OBD driver; http://www.lustre.org/ [14642.833198] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [14651.514979] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14684.597660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14684.625296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14685.849686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14685.935635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14685.990341] Lustre: lustre-MDT0000: new disk, initializing [14686.081083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14686.091465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14690.040566] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14699.399822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14699.479796] Lustre: 523050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14699.573329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14699.576934] Lustre: Skipped 1 previous similar message [14699.644365] Lustre: lustre-MDT0001: new disk, initializing [14699.781408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14699.829046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14699.844451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14703.659318] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14711.612557] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14716.518616] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14722.365603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14722.553601] Lustre: lustre-OST0000: new disk, initializing [14722.556987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14722.562229] Lustre: 525019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14722.639806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14728.260911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14728.272163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14728.475140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14729.057970] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [14736.879864] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14740.463551] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14740.629614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14744.300264] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14744.436792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14749.155388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14749.166264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14749.170164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14749.210954] Lustre: Skipped 1 previous similar message [14751.932921] Lustre: server umount lustre-OST0000 complete [14756.320338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14756.326941] LustreError: Skipped 1 previous similar message [14756.331890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14756.342598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14756.346858] Lustre: Skipped 1 previous similar message [14759.393511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14759.394390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14761.269960] Lustre: server umount lustre-MDT0000 complete [14764.067129] LustreError: 523043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786180597 with bad export cookie 465807766538639930 [14764.079341] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14764.295527] Lustre: server umount lustre-MDT0001 complete [14785.050492] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [14793.713928] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [14817.350035] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14520 sec ========= 05:17:27 (1786180647) [14818.977462] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:17:29 (1786180649) === [14822.451276] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:17:32 (1786180652) === [14853.782612] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [14855.564326] Key type lgssc unregistered [14855.773755] LNet: 531664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14855.778278] LNetError: 531664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14855.789838] LNet: Removed LNI 192.168.203.157@tcp [14856.407867] Key type .llcrypt unregistered [14856.409855] Key type ._llcrypt unregistered