[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456896722 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003221] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.023266] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028642] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] *** VALIDATE tmpfs *** [ 0.032485] *** VALIDATE proc *** [ 0.033262] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036028] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044542] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.046167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047678] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.056225] rcu: Hierarchical SRCU implementation. [ 0.058474] smp: Bringing up secondary CPUs ... [ 0.059603] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063593] smp: Brought up 1 node, 4 CPUs [ 0.065017] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.243599] node 0 deferred pages initialised in 174ms [ 0.246144] devtmpfs: initialized [ 0.247166] x86/mm: Memory block size: 128MB [ 0.249802] gcov: version magic: 0x41383552 [ 0.250571] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256393] pinctrl core: initialized pinctrl subsystem [ 0.259225] [ 0.259882] ************************************************************* [ 0.262017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265020] ** ** [ 0.268021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.271022] ** ** [ 0.273019] ** This means that this kernel is built to expose internal ** [ 0.276019] ** IOMMU data structures, which may compromise security on ** [ 0.280022] ** your system. ** [ 0.283019] ** ** [ 0.285018] ** If you see this message and you are not debugging the ** [ 0.288020] ** kernel, report this immediately to your vendor! ** [ 0.291021] ** ** [ 0.294032] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.298023] ************************************************************* [ 0.301769] NET: Registered protocol family 16 [ 0.303492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.306101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.309085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.313635] cpuidle: using governor menu [ 0.314492] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.317752] PCI: Using configuration type 1 for base access [ 0.320129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.330112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.338035] cryptd: max_cpu_qlen set to 1000 [ 0.341043] ACPI: Added _OSI(Module Device) [ 0.342027] ACPI: Added _OSI(Processor Device) [ 0.343017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.351811] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.358459] ACPI: Interpreter enabled [ 0.361067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362021] ACPI: Using IOAPIC for interrupt routing [ 0.364127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.368385] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.378597] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.380041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.383020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.386077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.391289] acpiphp: Slot [2] registered [ 0.393159] acpiphp: Slot [5] registered [ 0.394141] acpiphp: Slot [6] registered [ 0.396133] acpiphp: Slot [7] registered [ 0.397106] acpiphp: Slot [8] registered [ 0.398102] acpiphp: Slot [9] registered [ 0.399122] acpiphp: Slot [10] registered [ 0.401113] acpiphp: Slot [3] registered [ 0.402100] acpiphp: Slot [4] registered [ 0.404094] acpiphp: Slot [11] registered [ 0.405103] acpiphp: Slot [12] registered [ 0.406098] acpiphp: Slot [13] registered [ 0.408093] acpiphp: Slot [14] registered [ 0.409106] acpiphp: Slot [15] registered [ 0.410091] acpiphp: Slot [16] registered [ 0.412109] acpiphp: Slot [17] registered [ 0.413107] acpiphp: Slot [18] registered [ 0.414093] acpiphp: Slot [19] registered [ 0.416099] acpiphp: Slot [20] registered [ 0.417092] acpiphp: Slot [21] registered [ 0.418101] acpiphp: Slot [22] registered [ 0.420091] acpiphp: Slot [23] registered [ 0.421128] acpiphp: Slot [24] registered [ 0.423160] acpiphp: Slot [25] registered [ 0.425315] acpiphp: Slot [26] registered [ 0.427176] acpiphp: Slot [27] registered [ 0.429208] acpiphp: Slot [28] registered [ 0.431483] acpiphp: Slot [29] registered [ 0.433146] acpiphp: Slot [30] registered [ 0.435158] acpiphp: Slot [31] registered [ 0.437140] PCI host bridge to bus 0000:00 [ 0.439027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.442039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.451046] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.455033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.457261] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.460336] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.464525] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.474710] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.479963] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.483026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.486035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.489027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.493060] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.496020] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.500094] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.502683] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.508016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.520032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.526017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.533357] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.540015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.547013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.561016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.573096] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.579022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.588020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.605018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.616191] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.622017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.630019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.644015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.655706] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.663022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.670022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.687022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.703197] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.709017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.716013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.749673] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.758019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.765022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.784025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.795151] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.798424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.801376] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.803330] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.806246] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.811132] iommu: Default domain type: Passthrough [ 0.812483] SCSI subsystem initialized [ 0.814149] ACPI: bus type USB registered [ 0.816191] usbcore: registered new interface driver usbfs [ 0.818096] usbcore: registered new interface driver hub [ 0.820197] usbcore: registered new device driver usb [ 0.823191] pps_core: LinuxPPS API ver. 1 registered [ 0.825010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.829064] PTP clock support registered [ 0.831181] EDAC MC: Ver: 3.0.0 [ 0.833486] PCI: Using ACPI for IRQ routing [ 0.834525] NetLabel: Initializing [ 0.835000] NetLabel: domain hash size = 128 [ 0.837026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.839101] NetLabel: unlabeled traffic allowed by default [ 0.842132] vgaarb: loaded [ 0.843365] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.852004] clocksource: Switched to clocksource kvm-clock [ 0.959153] VFS: Disk quotas dquot_6.6.0 [ 0.961325] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.963846] *** VALIDATE ramfs *** [ 0.965084] *** VALIDATE hugetlbfs *** [ 0.966581] pnp: PnP ACPI init [ 0.968808] pnp: PnP ACPI: found 6 devices [ 0.989425] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.992088] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994082] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.996404] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.999217] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.001871] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.005111] NET: Registered protocol family 2 [ 1.009645] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.014789] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.018633] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.025559] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029278] TCP: Hash tables configured (established 65536 bind 65536) [ 1.032257] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.034799] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037683] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040930] NET: Registered protocol family 1 [ 1.043487] RPC: Registered named UNIX socket transport module. [ 1.046011] RPC: Registered udp transport module. [ 1.047721] RPC: Registered tcp transport module. [ 1.049235] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.052222] NET: Registered protocol family 44 [ 1.054372] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.056670] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.059107] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.061567] PCI: CLS 0 bytes, default 64 [ 1.063523] Unpacking initramfs... [ 2.501646] debug: unmapping init [mem 0xffff8f7c7cc54000-0xffff8f7c7ffbffff] [ 2.505925] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.508228] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.511587] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.008985] Initialise system trusted keyrings [ 3.010886] Key type blacklist registered [ 3.012824] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.021743] zbud: loaded [ 3.024814] *** VALIDATE nfs *** [ 3.025859] *** VALIDATE nfs4 *** [ 3.027606] pstore: using deflate compression [ 3.031813] Platform Keyring initialized [ 3.144479] NET: Registered protocol family 38 [ 3.146480] Key type asymmetric registered [ 3.148013] Asymmetric key parser 'x509' registered [ 3.149653] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.152897] io scheduler mq-deadline registered [ 3.154369] io scheduler kyber registered [ 3.156267] io scheduler bfq registered [ 3.158347] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.161704] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.164693] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.167616] ACPI: Power Button [PWRF] [ 3.173204] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.181188] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.197159] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.204614] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.222784] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.249852] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277834] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.281969] Non-volatile memory driver v1.3 [ 3.283129] Linux agpgart interface v0.103 [ 3.309658] virtio_blk virtio1: [vda] 147000 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.312242] vda: detected capacity change from 0 to 75264000 [ 3.327383] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.330407] vdb: detected capacity change from 0 to 1073741824 [ 3.346413] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.349373] vdc: detected capacity change from 0 to 2621440000 [ 3.364105] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.367429] vdd: detected capacity change from 0 to 2621440000 [ 3.388988] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392622] vde: detected capacity change from 0 to 4294967296 [ 3.409834] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.413295] vdf: detected capacity change from 0 to 4294967296 [ 3.422835] libphy: Fixed MDIO Bus: probed [ 3.427378] usbcore: registered new interface driver usbserial_generic [ 3.429712] usbserial: USB Serial support registered for generic [ 3.432042] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.436193] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.437967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.440327] mousedev: PS/2 mouse device common for all mice [ 3.443464] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.443919] rtc_cmos 00:05: RTC can wake from S4 [ 3.453292] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.457181] rtc_cmos 00:05: registered as rtc0 [ 3.464713] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.466123] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.466171] intel_pstate: CPU model not supported [ 3.472480] hid: raw HID events driver (C) Jiri Kosina [ 3.477085] usbcore: registered new interface driver usbhid [ 3.479136] usbhid: USB HID core driver [ 3.480713] drop_monitor: Initializing network drop monitor service [ 3.483283] Initializing XFRM netlink socket [ 3.485273] NET: Registered protocol family 10 [ 3.488393] Segment Routing with IPv6 [ 3.489985] NET: Registered protocol family 17 [ 3.492691] mpls_gso: MPLS GSO support [ 3.499981] RAS: Correctable Errors collector initialized. [ 3.501914] AVX version of gcm_enc/dec engaged. [ 3.503516] AES CTR mode by8 optimization enabled [ 3.585041] sched_clock: Marking stable (3584991167, 0)->(4482048909, -897057742) [ 3.589159] registered taskstats version 1 [ 3.591640] Loading compiled-in X.509 certificates [ 3.594206] zswap: loaded using pool lzo/zbud [ 3.621337] Key type big_key registered [ 3.633612] Key type encrypted registered [ 3.635132] ima: No TPM chip found, activating TPM-bypass! [ 3.637103] ima: Allocated hash algorithm: sha1 [ 3.638644] ima: No architecture policies found [ 3.640258] evm: Initialising EVM extended attributes: [ 3.642013] evm: security.selinux [ 3.643288] evm: security.ima [ 3.644331] evm: security.capability [ 3.645561] evm: HMAC attrs: 0x1 [ 3.648759] rtc_cmos 00:05: setting system clock to 2026-08-27 20:21:50 UTC (1787862110) [ 3.655175] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.658259] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.667092] Write protecting the kernel read-only data: 28672k [ 3.670858] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.673603] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.711665] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.721934] systemd[1]: Detected virtualization kvm. [ 3.724164] systemd[1]: Detected architecture x86-64. [ 3.726267] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.761959] systemd[1]: No hostname configured. [ 3.763827] systemd[1]: Set hostname to . [ 3.766366] random: systemd: uninitialized urandom read (16 bytes read) [ 3.769001] systemd[1]: Initializing machine ID from random generator. [ 3.816527] random: ln: uninitialized urandom read (6 bytes read) [ 3.900745] random: systemd: uninitialized urandom read (16 bytes read) [ 3.903681] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.908113] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.912918] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.484739] device-mapper: uevent: version 1.0.3 [ 4.486890] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.257675] virtio_net virtio0 ens2: renamed from eth0 [ 5.264659] random: fast init done [ 5.401791] scsi host0: ata_piix [ 5.410062] scsi host1: ata_piix [ 5.411851] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.414769] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.657887] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.152261] random: crng init done [ 10.154084] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.535438] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.631302] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.872922] SELinux: Disabled at runtime. [ 11.932257] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.941645] systemd[1]: Detected virtualization kvm. [ 11.943664] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.425618] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.429442] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.438281] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.442447] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.445605] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.452442] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.456915] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ 12.581324] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.862792] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.101849] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.155399] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.243243] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.264143] EDAC sbridge: Ver: 1.1.2 [ 14.748485] Key type dns_resolver registered [ 15.068922] NFS: Registering the id_resolver key type [ 15.071153] Key type id_resolver registered [ 15.073058] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 40.857080] libcfs: loading out-of-tree module taints kernel. [ 40.879076] Key type ._llcrypt registered [ 40.882855] Key type .llcrypt registered [ 40.932943] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 55.873570] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 57.269605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.289323] alg: No test for adler32 (adler32-zlib) [ 58.808688] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 59.742594] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 61.544499] Key type lgssc registered [ 63.125590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.435012] hrtimer: interrupt took 4485905 ns [ 78.081871] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.196692] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 127.756665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.804280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.988556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.013825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.089456] Lustre: lustre-MDT0000: new disk, initializing [ 129.171752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.197914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.556502] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.101697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.223570] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 147.276896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 147.284394] Lustre: Skipped 1 previous similar message [ 147.406394] Lustre: lustre-MDT0001: new disk, initializing [ 147.526516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 147.563462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 147.580171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 153.329341] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.414779] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.951167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 168.163201] Lustre: lustre-OST0000: new disk, initializing [ 168.167275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.172921] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.233970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.269364] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.227089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.243048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.303875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.321845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.493629] Lustre: lustre-OST0001: new disk, initializing [ 190.497849] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 190.504267] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.575730] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 197.695578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 197.719265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 197.845702] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 199.720638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.978288] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.493753] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.367035] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.957292] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 246.786188] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 250.184177] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 253.640778] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 256.210854] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 16:26:00 EDT 2026 [ 274.799341] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 276.805406] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.393660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.398575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.424657] Lustre: Skipped 3 previous similar messages [ 295.457298] Lustre: Skipped 3 previous similar messages [ 299.695139] Lustre: server umount lustre-MDT0000 complete [ 303.281286] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862410 with bad export cookie 10457881165984748907 [ 303.286924] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 303.302204] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 305.632575] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.633847] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 305.637191] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.656746] Lustre: Skipped 1 previous similar message [ 305.687021] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 308.640137] LustreError: 11330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.640573] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.666983] LustreError: 11330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 308.668610] Lustre: Skipped 2 previous similar messages [ 309.682492] Lustre: server umount lustre-MDT0001 complete [ 313.917073] Lustre: server umount lustre-OST0000 complete [ 317.400815] Lustre: server umount lustre-OST0001 complete [ 324.519848] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 333.909246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 372.795215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 373.008258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 373.028610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 373.098253] Lustre: lustre-MDT0000: new disk, initializing [ 373.173496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.194390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.784669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 386.387862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.494643] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.570893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.574330] Lustre: Skipped 1 previous similar message [ 386.681291] Lustre: lustre-MDT0001: new disk, initializing [ 386.823641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.857733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.864447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.473611] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 399.958982] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 405.666307] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 412.768202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 413.186984] Lustre: lustre-OST0000: new disk, initializing [ 413.195782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 413.203860] Lustre: 17051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 413.362290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.889800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 414.899170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 415.007101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.838422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 429.978524] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 435.741710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.069803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 440.750225] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 440.976264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 445.926496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.928119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.946719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 445.952935] Lustre: Skipped 1 previous similar message [ 445.960540] LustreError: Skipped 1 previous similar message [ 449.024530] Lustre: server umount lustre-OST0000 complete [ 453.603596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 453.616104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.629993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 453.634090] Lustre: Skipped 1 previous similar message [ 458.545400] Lustre: server umount lustre-MDT0000 complete [ 461.281261] LustreError: 15964:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.309198] LustreError: 15964:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 462.372386] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862569 with bad export cookie 10457881165984750440 [ 462.380822] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.381747] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 462.760946] Lustre: server umount lustre-MDT0001 complete [ 467.287838] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:29:32 (1787862572) [ 469.603434] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 471.572580] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:29:36 (1787862576) [ 477.855556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 478.285359] LustreError: 19462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.365556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.004526] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 483.808937] LustreError: 19463:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.182904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.244253] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 503.349897] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.782121] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.135887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.531456] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 528.422646] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.057593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.063861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.073754] Lustre: Skipped 1 previous similar message [ 537.084243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.097532] Lustre: Skipped 2 previous similar messages [ 539.104139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.758286] Lustre: server umount lustre-OST0000 complete [ 546.784781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 546.792249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.809171] Lustre: Skipped 1 previous similar message [ 551.473168] Lustre: server umount lustre-MDT0000 complete [ 554.470768] LustreError: 19458:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.496310] LustreError: 19458:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 555.060995] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862661 with bad export cookie 10457881165984751322 [ 555.066578] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.069550] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 555.366175] Lustre: server umount lustre-MDT0001 complete [ 567.041560] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 569.838422] Key type lgssc unregistered [ 570.145362] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.153039] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.181682] LNet: Removed LNI 192.168.203.144@tcp [ 571.104089] Key type .llcrypt unregistered [ 571.109682] Key type ._llcrypt unregistered [ 590.945184] Key type ._llcrypt registered [ 590.948307] Key type .llcrypt registered [ 591.059738] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 591.965298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 591.978605] alg: No test for adler32 (adler32-zlib) [ 593.108174] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 593.417708] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 595.098817] Key type lgssc registered [ 596.083987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 604.135730] loop: module loaded [ 604.207065] loop0: detected capacity change from 0 to 204800000 [ 604.416511] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 604.434980] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 604.698098] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.364199] loop0: detected capacity change from 0 to 204800000 [ 610.540194] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.553839] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 610.574649] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.657634] loop0: detected capacity change from 0 to 204800000 [ 619.113234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 619.135464] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 620.402450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 620.474896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 620.563733] Lustre: lustre-MDT0000: new disk, initializing [ 620.627732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.639841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 625.159471] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 633.041764] loop1: detected capacity change from 0 to 204800000 [ 638.201222] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 638.472629] Lustre: lustre-OST07c6: new disk, initializing [ 638.483336] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 638.491594] Lustre: Skipped 1 previous similar message [ 638.500640] Lustre: 26164:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 638.576815] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 644.182544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 644.187954] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 644.221840] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 644.253919] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 648.558588] Lustre: 25151:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 648.569928] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 648.581025] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 648.586974] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 648.598289] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 648.604627] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 665.145790] Lustre: 25149:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 665.158757] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 665.172664] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.185929] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 665.198330] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 665.213945] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.055196] Lustre: 25149:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 670.066352] Lustre: 25149:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 670.071695] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 670.077215] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.083309] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 670.089211] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.095059] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 670.104947] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.116543] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 670.128702] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 670.134111] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 670.140660] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.356867] Lustre: 26500:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 673.371601] Lustre: 26500:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.382634] Lustre: 26500:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 673.390280] Lustre: 26500:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.395282] Lustre: 26500:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.407571] Lustre: 26500:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.416861] Lustre: 26500:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 673.429240] Lustre: 26500:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.438155] Lustre: 26500:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.444448] Lustre: 26500:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.454769] Lustre: 26500:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.464458] Lustre: 26500:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.336756] Lustre: 26500:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 678.355369] Lustre: 26500:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 678.365810] Lustre: 26500:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 678.386276] Lustre: 26500:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.389907] Lustre: 26500:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 678.393873] Lustre: 26500:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.397375] Lustre: 26500:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 678.407287] Lustre: 26500:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.411827] Lustre: 26500:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 678.417292] Lustre: 26500:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 678.421567] Lustre: 26500:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 678.426020] Lustre: 26500:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.298652] Lustre: 25150:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 689.307135] Lustre: 25150:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 689.315026] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.322012] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 689.337782] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.345805] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 689.354732] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 689.366158] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 689.384511] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 689.389169] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 689.394164] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.401479] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 741.344790] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 741.355511] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.373730] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 743.790590] Lustre: server umount lustre-OST07c6 complete [ 752.904141] Lustre: server umount lustre-MDT0000 complete [ 768.858055] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 771.387478] Key type lgssc unregistered [ 771.704242] LNet: 28724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.713884] LNetError: 28724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.735603] LNet: Removed LNI 192.168.203.144@tcp [ 772.556350] Key type .llcrypt unregistered [ 772.559221] Key type ._llcrypt unregistered [ 790.271374] Key type ._llcrypt registered [ 790.275224] Key type .llcrypt registered [ 790.356670] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 791.232976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.307063] alg: No test for adler32 (adler32-zlib) [ 792.325891] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 792.551124] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 794.167924] Key type lgssc registered [ 795.142984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 801.539882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 801.567557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 802.987631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.450742] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 812.512904] LustreError: 29997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 812.532556] LustreError: 29997:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 816.238389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.479306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 821.013591] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 829.007711] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 834.183898] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 839.611924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 839.916833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.603519] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 848.109768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 852.220862] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 867.816268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 867.826191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 867.848200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.832351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 868.845931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 868.849200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.853866] LustreError: Skipped 1 previous similar message [ 868.885037] Lustre: Skipped 1 previous similar message [ 872.274918] Lustre: server umount lustre-OST0000 complete [ 878.049738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.053914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.071449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.991957] Lustre: server umount lustre-MDT0000 complete [ 884.197429] LustreError: 29996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.213446] LustreError: 29996:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 885.306068] LustreError: 32777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862992 with bad export cookie 12205150621396332530 [ 885.309394] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.316429] LustreError: 32777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 885.543771] Lustre: server umount lustre-MDT0001 complete [ 894.436753] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 896.788736] Key type lgssc unregistered [ 897.057748] LNet: 33732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.063472] LNetError: 33732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.092439] LNet: Removed LNI 192.168.203.144@tcp [ 897.934743] Key type .llcrypt unregistered [ 897.939538] Key type ._llcrypt unregistered [ 914.534543] Key type ._llcrypt registered [ 914.539532] Key type .llcrypt registered [ 914.648660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 915.824634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.918574] alg: No test for adler32 (adler32-zlib) [ 917.003665] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 917.283448] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 918.975161] Key type lgssc registered [ 920.066179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 927.633847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 927.654763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 929.116230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.164462] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 938.466761] LustreError: 34986:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 938.490215] LustreError: 34986:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 940.931558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.311771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 945.843605] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 952.710655] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 956.991381] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 962.923102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 963.336042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.577609] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 973.575592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 975.337493] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 987.107208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.121624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.134179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.153973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.157599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.190611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.265125] Lustre: server umount lustre-OST0000 complete [ 997.859471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 997.886586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 997.911367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.927952] Lustre: Skipped 1 previous similar message [ 1002.939702] Lustre: server umount lustre-MDT0000 complete [ 1004.514931] LustreError: 35715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.557441] LustreError: 35715:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1007.029611] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863113 with bad export cookie 201365732380115893 [ 1007.040723] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.043997] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1007.324834] Lustre: server umount lustre-MDT0001 complete [ 1019.548426] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1022.385965] Key type lgssc unregistered [ 1022.670811] LNet: 38522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.680031] LNetError: 38522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.697430] LNet: Removed LNI 192.168.203.144@tcp [ 1023.674281] Key type .llcrypt unregistered [ 1023.677507] Key type ._llcrypt unregistered [ 1052.632410] Key type ._llcrypt registered [ 1052.634280] Key type .llcrypt registered [ 1052.796503] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1053.712269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.726261] alg: No test for adler32 (adler32-zlib) [ 1054.791782] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1055.045397] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1056.711163] Key type lgssc registered [ 1057.809974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.502806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.535970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.050843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1070.428420] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1075.170932] LustreError: 39938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.193901] LustreError: 39938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1079.216365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.576222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.754032] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1091.412403] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1097.118728] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.158292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.295926] LustreError: 41759:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1104.501309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.248701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1111.029741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1117.712019] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1123.817784] Lustre: server umount lustre-MDT0000 complete [ 1124.860688] LustreError: 39934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.861438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.880790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.193784] LustreError: 39917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863235 with bad export cookie 2179042552799371863 [ 1128.205910] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.950553] LustreError: 41372:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.965868] LustreError: 41372:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1129.976103] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [ 1131.489326] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.502168] Lustre: Skipped 2 previous similar messages [ 1131.509713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1133.537100] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1134.686637] Lustre: server umount lustre-MDT0001 complete [ 1156.879766] Lustre: server umount lustre-OST0000 complete [ 1175.254440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1177.573904] Key type lgssc unregistered [ 1177.900708] LNet: 43805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.907963] LNetError: 43805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.924861] LNet: Removed LNI 192.168.203.144@tcp [ 1178.710285] Key type .llcrypt unregistered [ 1178.712966] Key type ._llcrypt unregistered [ 1207.400651] Key type ._llcrypt registered [ 1207.402277] Key type .llcrypt registered [ 1207.599144] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1208.665887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.720802] alg: No test for adler32 (adler32-zlib) [ 1209.908441] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1210.220984] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1211.839854] Key type lgssc registered [ 1213.732807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.932646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.986343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.606296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.342469] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1235.938477] LustreError: 45223:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.957644] LustreError: 45223:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1239.420785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.765231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1244.502492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1250.890801] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1256.596508] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1263.917904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.441746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.596390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1270.944885] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1278.324754] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1286.112569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.135194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.135694] Lustre: Skipped 1 previous similar message [ 1286.151969] Lustre: Skipped 1 previous similar message [ 1291.242902] Lustre: lustre-OST0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 1291.246596] Lustre: Skipped 1 previous similar message [ 1291.778406] Lustre: server umount lustre-OST0000 complete [ 1311.715241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.717528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.730831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.739762] Lustre: Skipped 1 previous similar message [ 1311.747712] Lustre: Skipped 1 previous similar message [ 1316.833622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1316.843704] Lustre: Skipped 1 previous similar message [ 1317.100579] Lustre: server umount lustre-MDT0000 complete [ 1317.857400] LustreError: 45220:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.877848] LustreError: 45220:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1320.852666] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863427 with bad export cookie 11241205057295334333 [ 1320.855672] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.864068] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1335.263865] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1335.582423] Lustre: server umount lustre-MDT0001 complete [ 1346.617928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1349.250800] Key type lgssc unregistered [ 1349.602303] LNet: 48940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.610788] LNetError: 48940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.625269] LNet: Removed LNI 192.168.203.144@tcp [ 1350.433531] Key type .llcrypt unregistered [ 1350.438412] Key type ._llcrypt unregistered [ 1380.624246] Key type ._llcrypt registered [ 1380.627452] Key type .llcrypt registered [ 1380.752699] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1382.286111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1382.317727] alg: No test for adler32 (adler32-zlib) [ 1383.481692] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1383.747261] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1385.425589] Key type lgssc registered [ 1386.489540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1394.178676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1394.194821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.564299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1400.402583] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1405.922384] LustreError: 50360:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.947827] LustreError: 50360:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1409.568361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.091516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.788686] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1423.183682] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.535466] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1437.740180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.124908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.191971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1445.255881] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1453.986117] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.410271] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1463.717713] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1476.243841] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2716h46m20s [ 1478.270759] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1481.222608] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 1481.696463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1481.704412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.724985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.800865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.811140] Lustre: Skipped 5 previous similar messages [ 1487.339177] LustreError: 50354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1487.481741] Lustre: server umount lustre-MDT0000 complete [ 1488.871965] LustreError: 50355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.882287] LustreError: 50355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1491.557557] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1493.492585] LustreError: 50354:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.551548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.682603] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1498.880235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.956583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1499.976565] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1503.396850] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1506.864629] Lustre: DEBUG MARKER: Wait for df (36791) ... [ 1509.613087] Lustre: DEBUG MARKER: done [ 1515.491596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.517406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1515.536568] Lustre: Skipped 3 previous similar messages [ 1515.566724] Lustre: Skipped 1 previous similar message [ 1520.611536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.742891] Lustre: server umount lustre-OST0000 complete [ 1530.851575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1530.852114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1530.878205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1530.878213] Lustre: Skipped 1 previous similar message [ 1530.932770] Lustre: Skipped 1 previous similar message [ 1532.823980] Lustre: server umount lustre-MDT0000 complete [ 1537.804303] LustreError: 50338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863644 with bad export cookie 1299225487027049972 [ 1537.816791] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1537.836248] LustreError: 50338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1538.347302] Lustre: server umount lustre-MDT0001 complete [ 1550.408925] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1554.460409] Key type lgssc unregistered [ 1554.915293] LNet: 55286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.929249] LNetError: 55286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.959823] LNet: Removed LNI 192.168.203.144@tcp [ 1556.287045] Key type .llcrypt unregistered [ 1556.291837] Key type ._llcrypt unregistered [ 1598.281758] Key type ._llcrypt registered [ 1598.287097] Key type .llcrypt registered [ 1598.412294] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1599.462022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.496163] alg: No test for adler32 (adler32-zlib) [ 1600.595459] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1600.942502] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1602.719163] Key type lgssc registered [ 1603.845433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1612.641876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1612.663795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.048505] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1614.092543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1614.229530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.541857] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1627.612100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.785565] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1627.828400] Lustre: 57498:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1628.134020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1632.734044] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1640.835413] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1646.381735] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1653.222186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.480968] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1653.645190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.971530] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1660.053983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1667.318799] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.829332] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2716h49m38s [ 1675.545103] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1683.119996] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1685.594129] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1685.641917] LustreError: 57504:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1706.010618] Lustre: lustre-MDT0000: Client fda24a6a-2f8d-4f60-a3a9-2cb9b0922549 (at 192.168.203.44@tcp) reconnecting [ 1725.743123] LustreError: 57504:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1727.926375] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1730.535604] Lustre: DEBUG MARKER: done [ 1736.672742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1736.679633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.690872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.190705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1737.202503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.213219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1737.220402] Lustre: Skipped 1 previous similar message [ 1738.702652] Lustre: server umount lustre-OST0000 complete [ 1745.894828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.904066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.929359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1749.026304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1749.038452] Lustre: Skipped 4 previous similar messages [ 1755.118589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.130595] Lustre: Skipped 1 previous similar message [ 1757.151148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1757.430872] Lustre: server umount lustre-MDT0000 complete [ 1761.257174] LustreError: 57509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.273590] LustreError: 57509:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1761.725609] LustreError: 60261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863868 with bad export cookie 10888022668455016286 [ 1761.732813] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1761.742962] LustreError: 60261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1761.975400] Lustre: server umount lustre-MDT0001 complete [ 1775.915508] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1779.011827] Key type lgssc unregistered [ 1779.307324] LNet: 61838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.314678] LNetError: 61838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.359118] LNet: Removed LNI 192.168.203.144@tcp [ 1780.315470] Key type .llcrypt unregistered [ 1780.318553] Key type ._llcrypt unregistered [ 1823.031355] Key type ._llcrypt registered [ 1823.034871] Key type .llcrypt registered [ 1823.146416] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1824.305486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.513909] alg: No test for adler32 (adler32-zlib) [ 1825.721133] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1826.125574] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1827.871345] Key type lgssc registered [ 1829.109797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1833.085221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1833.139958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.420900] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1834.443848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1834.586786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.722579] Lustre: Failing over lustre-MDT0000 [ 1836.149113] Lustre: server umount lustre-MDT0000 complete [ 1844.162416] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 16:52:28 (1787863948) [ 1851.705252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.073389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.087896] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1864.471162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.557990] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1864.596585] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1864.833154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.764262] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1877.600317] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1883.905894] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1891.053729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1891.297802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1891.565391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.162678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1899.425663] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1908.240636] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1915.028840] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1926.113696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1926.124907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1926.146904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.162955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.168801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.197043] Lustre: Skipped 1 previous similar message [ 1929.642129] Lustre: server umount lustre-MDT0000 complete [ 1933.527250] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864040 with bad export cookie 729115846700831162 [ 1933.529543] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.540421] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1933.934584] Lustre: server umount lustre-MDT0001 complete [ 1938.005319] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1942.913095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1951.214670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.783620] Lustre: 63561:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864045/real 1787864045] req@000000000a65aade x1874711210405120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1954.815534] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.948633] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa1e56ffe2b61a9d [ 1958.959653] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1959.315229] LustreError: 66487:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.334841] LustreError: 66487:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1959.400564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.519104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1960.416302] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864050/real 1787864050] req@0000000021d399f9 x1874711210405248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1963.432550] LustreError: 66488:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.175674] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1964.455755] LustreError: 68409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.468064] LustreError: 68409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1964.470088] Lustre: 63558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864055/real 1787864055] req@00000000fa91599e x1874711210405504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.571925] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864060/real 1787864060] req@00000000c389455e x1874711210405632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.649064] LustreError: 68407:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.665678] LustreError: 68407:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1971.342832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.656955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.211262] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1976.815638] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1983.292389] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1984.044357] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1989.008114] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.282766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1997.293170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1997.310618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1997.325024] Lustre: Skipped 2 previous similar messages [ 1999.328367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.605148] Lustre: server umount lustre-MDT0000 complete [ 2004.450273] LustreError: 69654:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.485533] LustreError: 69654:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2006.845184] LustreError: 65372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864113 with bad export cookie 729115846700833437 [ 2006.848700] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2006.863590] LustreError: 65372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2007.242214] Lustre: server umount lustre-MDT0001 complete [ 2013.797939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2022.245128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.759206] Lustre: 63558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864116/real 1787864116] req@0000000090a5e147 x1874711210433024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2025.788202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.800462] Lustre: Skipped 2 previous similar messages [ 2031.976205] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa1e56ffe2b62000 [ 2031.986788] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [ 2032.215862] LustreError: 66487:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2032.337953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.411597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2036.364670] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2037.409544] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864128/real 1787864128] req@0000000036ac844c x1874711210433664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787864144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.435765] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2045.083813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.707305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2050.535020] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2057.276403] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2063.454303] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.009435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.023104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.047468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.063048] Lustre: Skipped 2 previous similar messages [ 2076.886419] Lustre: server umount lustre-MDT0000 complete [ 2080.151763] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864187 with bad export cookie 729115846700834816 [ 2080.152150] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2080.168783] LustreError: 64626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2080.235285] LustreError: 71093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.243987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2080.247155] LustreError: 71093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2080.256851] Lustre: Skipped 3 previous similar messages [ 2080.481600] Lustre: server umount lustre-MDT0001 complete [ 2086.771660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2089.720772] Lustre: DEBUG MARKER: files compared the same [ 2091.471586] Lustre: server umount lustre-OST0000 complete [ 2106.943174] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2109.510315] Key type lgssc unregistered [ 2109.778375] LNet: 74369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.788579] LNetError: 74369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.820273] LNet: Removed LNI 192.168.203.144@tcp [ 2110.616682] Key type .llcrypt unregistered [ 2110.619584] Key type ._llcrypt unregistered [ 2133.999542] Key type ._llcrypt registered [ 2134.005487] Key type .llcrypt registered [ 2134.137816] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2135.204271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.237772] alg: No test for adler32 (adler32-zlib) [ 2136.258185] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2136.415387] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2138.055228] Key type lgssc registered [ 2139.274473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2146.119982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2146.165586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.631137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.122688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2158.051847] LustreError: 75787:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.068624] LustreError: 75787:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2161.725991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.106515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.051805] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2173.466481] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2178.035159] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2184.095586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2184.308301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.377758] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug +malloc all [ 2192.487390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2199.461165] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2212.319908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2212.332745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2212.353534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2213.348851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2213.349778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.349859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.353453] Lustre: Skipped 1 previous similar message [ 2218.464629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.467859] Lustre: Skipped 1 previous similar message [ 2223.071425] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2223.333799] Lustre: server umount lustre-OST0000 complete [ 2228.704370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.717283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.723282] Lustre: Skipped 1 previous similar message [ 2233.420312] Lustre: server umount lustre-MDT0000 complete [ 2233.823931] LustreError: 75781:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.837941] LustreError: 75781:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2237.204559] LustreError: 75766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864344 with bad export cookie 6728863625503985356 [ 2237.205249] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.209900] LustreError: 75766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2237.539425] Lustre: server umount lustre-MDT0001 complete [ 2249.883465] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2253.031553] Key type lgssc unregistered [ 2253.350541] LNet: 79325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.362876] LNetError: 79325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.403301] LNet: Removed LNI 192.168.203.144@tcp [ 2254.383534] Key type .llcrypt unregistered [ 2254.385073] Key type ._llcrypt unregistered [ 2278.118786] Key type ._llcrypt registered [ 2278.125831] Key type .llcrypt registered [ 2278.272186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2279.261994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2279.395577] alg: No test for adler32 (adler32-zlib) [ 2280.408273] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2280.578645] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2282.223191] Key type lgssc registered [ 2283.332951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2290.642806] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2290.670595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2306.207869] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2311.328264] LustreError: 80715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 2338.079939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.365920] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2352.418922] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2360.617725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.775760] Lustre: *** cfs_fail_loc=706, val=0*** [ 2361.389834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.509481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2365.940355] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2366.433175] LustreError: 80741:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.448586] LustreError: 80741:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2371.552720] LustreError: 80742:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.587211] LustreError: 81816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.618158] LustreError: 81816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2375.247430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.650349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.790617] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2387.969658] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.213244] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2397.155185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2397.161843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.184200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.249719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.254666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2401.255826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.372919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.377830] Lustre: Skipped 1 previous similar message [ 2410.976620] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2411.088502] Lustre: server umount lustre-OST0000 complete [ 2416.609586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.610495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2416.611024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.611028] Lustre: Skipped 1 previous similar message [ 2416.617399] Lustre: Skipped 1 previous similar message [ 2420.615692] Lustre: server umount lustre-MDT0000 complete [ 2421.733055] LustreError: 81816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.763346] LustreError: 81816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2423.975387] LustreError: 80722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864530 with bad export cookie 4543556290841452814 [ 2423.981619] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.000231] LustreError: 80722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2424.384796] Lustre: server umount lustre-MDT0001 complete [ 2435.651109] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2438.126180] Key type lgssc unregistered [ 2438.378712] LNet: 84352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.387672] LNetError: 84352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.399273] LNet: Removed LNI 192.168.203.144@tcp [ 2439.163396] Key type .llcrypt unregistered [ 2439.166460] Key type ._llcrypt unregistered [ 2465.400422] Key type ._llcrypt registered [ 2465.405806] Key type .llcrypt registered [ 2465.515546] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2466.288464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2466.300183] alg: No test for adler32 (adler32-zlib) [ 2467.417829] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2467.708720] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2469.448523] Key type lgssc registered [ 2470.394690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.912683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.942889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.064400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2493.792539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.326837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.743843] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2499.558199] LustreError: 86471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.576022] LustreError: 86471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2504.679510] LustreError: 86470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.703595] LustreError: 86470:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2507.247082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2507.592930] LustreError: 86471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.615821] LustreError: 86471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2507.637338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.540232] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2512.865312] LustreError: 86475:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.899176] LustreError: 86475:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2517.990779] LustreError: 86474:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.004232] LustreError: 86474:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2519.387257] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.402825] Lustre: MGS: Client 3c7d1df2-0622-433e-b45c-21f9e77b5ea5 (at 0@lo) reconnecting [ 2519.437973] LustreError: 86485:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2519.704994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2519.807943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2523.342113] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2536.433488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.444772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.446820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2541.540939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2541.556158] Lustre: Skipped 2 previous similar messages [ 2541.864820] Lustre: server umount lustre-OST0000 complete [ 2546.658438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.660417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.668818] Lustre: Skipped 2 previous similar messages [ 2546.688280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2551.173103] Lustre: server umount lustre-MDT0000 complete [ 2551.781130] LustreError: 88015:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.802940] LustreError: 88015:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2554.466244] Lustre: server umount lustre-MDT0001 complete [ 2557.678567] Lustre: server umount MGS complete [ 2568.019068] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:04:33 (1787864673) [ 2595.891802] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2604.993697] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2640.757191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.309482] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2654.623021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.904236] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2654.975370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2655.043112] Lustre: lustre-MDT0001: new disk, initializing [ 2655.139356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2655.176698] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2655.190929] Lustre: 93636:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2655.198967] LustreError: 93636:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2659.187124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2668.880783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2669.158336] Lustre: lustre-OST0000: new disk, initializing [ 2669.172701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2669.184189] Lustre: 94404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2669.284635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.891055] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2678.307740] Lustre: 94860:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000389f3c75 x1874711883585920/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787864801 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2683.925277] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2691.694892] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.707407] Lustre: MGS: Client a97401dc-0757-4ec3-b304-8b7ba8f4580b (at 0@lo) reconnecting [ 2691.761592] Lustre: 95450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2691.775678] LustreError: 93622:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2691.789085] LustreError: 93622:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2691.831554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2691.941531] Lustre: lustre-MDT0000: new disk, initializing [ 2692.105106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.140566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2692.345818] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2693.217849] Lustre: 95475:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000014088e48 x1874711883592704/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787864816 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2697.085923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2698.734285] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2702.388730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2702.406718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2702.541181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2728.416686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2728.436952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.466573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2728.477890] Lustre: Skipped 1 previous similar message [ 2730.473234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.728549] Lustre: server umount lustre-OST0000 complete [ 2740.708185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2740.714600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.725793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.734896] Lustre: Skipped 1 previous similar message [ 2740.754146] Lustre: Skipped 3 previous similar messages [ 2743.681755] Lustre: server umount lustre-MDT0000 complete [ 2745.824458] LustreError: 95728:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.841254] LustreError: 95728:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2754.075525] Lustre: server umount lustre-MDT0001 complete [ 2758.017243] Lustre: server umount MGS complete [ 2770.188714] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:07:54 (1787864874) [ 2788.266915] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2791.395469] Key type lgssc unregistered [ 2791.888876] LNet: 97974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.902984] LNetError: 97974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2791.916557] LNet: Removed LNI 192.168.203.144@tcp [ 2792.939469] Key type .llcrypt unregistered [ 2792.945255] Key type ._llcrypt unregistered [ 2811.055702] Key type ._llcrypt registered [ 2811.059308] Key type .llcrypt registered [ 2811.207874] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2812.252037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2812.373753] alg: No test for adler32 (adler32-zlib) [ 2813.467182] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2813.673413] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2815.359156] Key type lgssc registered [ 2816.658173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2824.741750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2824.771426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.396798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.742576] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2836.964471] LustreError: 99200:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.984456] LustreError: 99200:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2839.378353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.654708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.785987] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2851.800518] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2857.820151] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2865.603850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.048910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.204979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2874.412022] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2884.069623] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2894.819104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2894.824492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.845202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2894.902584] Lustre: Skipped 1 previous similar message [ 2899.745036] Lustre: server umount lustre-OST0000 complete [ 2904.062161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.076928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2904.083611] Lustre: Skipped 1 previous similar message [ 2904.110523] Lustre: Skipped 2 previous similar messages [ 2909.154373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2909.160280] Lustre: Skipped 1 previous similar message [ 2914.273817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2914.284787] Lustre: Skipped 1 previous similar message [ 2918.367115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2918.689100] Lustre: server umount lustre-MDT0000 complete [ 2919.397068] LustreError: 99201:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.416230] LustreError: 99201:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2923.317482] LustreError: 99180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865030 with bad export cookie 10877810647222397633 [ 2923.345277] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.897635] Lustre: server umount lustre-MDT0001 complete [ 2937.399967] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2941.115670] Key type lgssc unregistered [ 2941.426740] LNet: 102738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.449973] LNetError: 102738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2941.476464] LNet: Removed LNI 192.168.203.144@tcp [ 2942.652560] Key type .llcrypt unregistered [ 2942.658609] Key type ._llcrypt unregistered [ 2959.845122] Key type ._llcrypt registered [ 2959.847168] Key type .llcrypt registered [ 2959.943794] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2960.694503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2960.712945] alg: No test for adler32 (adler32-zlib) [ 2961.682463] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2961.975969] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2963.671741] Key type lgssc registered [ 2964.567245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2972.190582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2972.209393] LustreError: 103823:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2972.232140] LustreError: 103824:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2972.248321] LustreError: 103823:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 2972.266771] LustreError: 103823:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2972.287824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.294322] LustreError: 103823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2973.972115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.297253] LustreError: 103861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.322532] LustreError: 103861:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2982.721374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.196572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2987.857285] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2994.174149] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing lsmod [ 2997.029621] LustreError: 105023:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2997.083551] LustreError: 105032:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2997.091024] LustreError: 105023:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4943 [ 2997.101181] LustreError: 105023:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2997.112618] LustreError: 105023:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2997.117945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.703462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.727748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3003.873688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3003.880931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3003.894588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.898971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3004.899648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.911401] Lustre: Skipped 1 previous similar message [ 3004.932120] Lustre: Skipped 1 previous similar message [ 3006.987669] Lustre: server umount lustre-MDT0000 complete [ 3008.036728] LustreError: 103861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.064945] LustreError: 103861:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3011.048157] LustreError: 104300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865117 with bad export cookie 2878829618218659328 [ 3011.055133] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3011.059927] LustreError: 104300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3012.192907] LustreError: 104293:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000af90a45e x1874712401565184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3012.345690] Lustre: server umount lustre-MDT0001 complete [ 3026.354471] Lustre: server umount lustre-OST0000 complete [ 3035.403750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.958513] LustreError: 106095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.039063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.265357] LustreError: 106096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.292201] LustreError: 106096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3041.330718] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3045.349431] LustreError: 106095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.737317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.065213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3056.431691] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3065.221796] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.479230] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3079.744219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.261498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.281276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3087.445142] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3095.796823] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.770712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.774574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.792852] Lustre: Skipped 1 previous similar message [ 3105.795060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3108.730087] Lustre: server umount lustre-OST0000 complete [ 3116.009472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.012338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.042605] Lustre: Skipped 1 previous similar message [ 3116.069134] Lustre: Skipped 1 previous similar message [ 3119.061587] Lustre: server umount lustre-MDT0000 complete [ 3121.121300] LustreError: 108589:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.155973] LustreError: 108589:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3123.196702] LustreError: 107940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865230 with bad export cookie 2878829618218660259 [ 3123.200104] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.221896] LustreError: 107940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3123.719497] Lustre: server umount lustre-MDT0001 complete [ 3137.355666] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3140.454741] Key type lgssc unregistered [ 3140.758079] LNet: 109633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.762834] LNetError: 109633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.798745] LNet: Removed LNI 192.168.203.144@tcp [ 3141.536937] Key type .llcrypt unregistered [ 3141.540665] Key type ._llcrypt unregistered [ 3166.033387] Key type ._llcrypt registered [ 3166.035716] Key type .llcrypt registered [ 3166.158036] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3167.172372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3167.213125] alg: No test for adler32 (adler32-zlib) [ 3168.324701] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 3168.580308] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3170.266555] Key type lgssc registered [ 3171.705853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.080676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.118442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.702501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.259603] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3192.306692] LustreError: 111053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.329793] LustreError: 111053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3196.422415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.015180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.665582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3211.470937] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3219.585237] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3227.835745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3228.494386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.510079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3234.758381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3242.717461] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3247.832914] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3251.985080] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3258.336541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.348053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.356184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.366471] Lustre: Skipped 1 previous similar message [ 3258.390953] Lustre: Skipped 1 previous similar message [ 3262.464930] Lustre: server umount lustre-OST0000 complete [ 3268.578416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.590382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3268.598521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.897046] Lustre: server umount lustre-MDT0000 complete [ 3273.696099] LustreError: 111054:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.712985] LustreError: 111054:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3276.963139] LustreError: 113583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865383 with bad export cookie 17531555064387642942 [ 3276.966475] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3276.976265] LustreError: 113583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3277.644129] Lustre: server umount lustre-MDT0001 complete [ 3289.903281] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3292.689374] Key type lgssc unregistered [ 3293.012033] LNet: 114694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3293.019291] LNetError: 114694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3293.039305] LNet: Removed LNI 192.168.203.144@tcp [ 3293.905539] Key type .llcrypt unregistered [ 3293.908396] Key type ._llcrypt unregistered [ 3311.237512] Key type ._llcrypt registered [ 3311.241749] Key type .llcrypt registered [ 3311.364812] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3312.797661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3312.820256] alg: No test for adler32 (adler32-zlib) [ 3313.902144] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 3314.204832] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3315.853092] Key type lgssc registered [ 3317.325679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3328.423586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3328.461029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.170232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3336.367027] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3339.745756] LustreError: 115918:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.765078] LustreError: 115918:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3344.869265] LustreError: 115919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.335684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.816263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3352.078929] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3360.646770] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3367.373974] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3374.555159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3375.051300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.320988] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3389.378189] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3399.621874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3407.841446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3407.850544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.872157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.890498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.897859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.922016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.944860] Lustre: Skipped 1 previous similar message [ 3413.490305] Lustre: server umount lustre-OST0000 complete [ 3418.601146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3418.608957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.626927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.770859] Lustre: server umount lustre-MDT0000 complete [ 3425.252797] LustreError: 117104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3425.277516] LustreError: 117104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3427.528664] LustreError: 115900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865534 with bad export cookie 17641853992828313147 [ 3427.531438] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.549487] LustreError: 115900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3428.085851] Lustre: server umount lustre-MDT0001 complete [ 3440.335694] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3443.252274] Key type lgssc unregistered [ 3443.567690] LNet: 119477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3443.575589] LNetError: 119477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3443.598452] LNet: Removed LNI 192.168.203.144@tcp [ 3444.513305] Key type .llcrypt unregistered [ 3444.514966] Key type ._llcrypt unregistered [ 3470.772794] Key type ._llcrypt registered [ 3470.774543] Key type .llcrypt registered [ 3470.939843] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3472.899370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3472.940786] alg: No test for adler32 (adler32-zlib) [ 3474.166593] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 3474.504056] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3476.287171] Key type lgssc registered [ 3477.946626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3486.615938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3486.632234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.057807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.533976] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3497.441732] LustreError: 120897:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3497.464160] LustreError: 120897:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3502.563124] LustreError: 120898:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.494177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.818996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.722852] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3516.677848] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3522.806573] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3529.754770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3530.066588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.167167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3537.485635] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3545.546913] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3555.179919] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3558.769303] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3558.777271] Lustre: Skipped 2 previous similar messages [ 3563.411535] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3563.418601] Lustre: Skipped 2 previous similar messages [ 3565.157964] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3565.184222] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3571.343690] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3571.357196] Lustre: Skipped 2 previous similar messages [ 3576.885222] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.44@tcp 192.168.0.[2,10]@tcp [ 3580.658413] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.44/24@tcp in log lustre-MDT0000 [ 3580.670407] Lustre: Skipped 2 previous similar messages [ 3586.122506] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.44/24@tcp [ 3586.125676] Lustre: Skipped 1 previous similar message [ 3593.190969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3593.193371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.195849] LustreError: Skipped 1 previous similar message [ 3593.215795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.720833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.734217] LustreError: Skipped 2 previous similar messages [ 3594.738681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.745644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.278949] Lustre: server umount lustre-OST0000 complete [ 3604.964977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.980764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3604.987844] Lustre: Skipped 2 previous similar messages [ 3608.852883] Lustre: server umount lustre-MDT0000 complete [ 3610.087655] LustreError: 120898:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.111327] LustreError: 120898:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3612.927905] LustreError: 120878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865719 with bad export cookie 11199696087092237819 [ 3612.936654] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3612.946594] LustreError: 120878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3613.419808] Lustre: server umount lustre-MDT0001 complete [ 3627.627794] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3630.952675] Key type lgssc unregistered [ 3631.317618] LNet: 125789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.323418] LNetError: 125789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3632.358227] LNet: Removed LNI 192.168.203.144@tcp [ 3633.697925] Key type .llcrypt unregistered [ 3633.699766] Key type ._llcrypt unregistered [ 3665.799975] Key type ._llcrypt registered [ 3665.802360] Key type .llcrypt registered [ 3665.904164] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3667.552299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3667.599273] alg: No test for adler32 (adler32-zlib) [ 3668.726834] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 3669.020172] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3670.839192] Key type lgssc registered [ 3672.387782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3680.857400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3680.886538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3682.394472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.408359] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3682.458546] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.44/24@tcp [ 3687.960889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3692.515945] LustreError: 127351:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.552897] LustreError: 127351:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3697.514512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.635835] LustreError: 127350:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.897409] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3697.926448] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.44/24@tcp [ 3697.974122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3703.306901] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3710.815665] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3718.003351] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3725.301245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3725.701192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.796723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3732.554091] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3741.178445] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3758.568323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3758.575085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.577123] LustreError: Skipped 1 previous similar message [ 3758.579146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.629869] Lustre: Skipped 1 previous similar message [ 3760.749843] Lustre: server umount lustre-OST0000 complete [ 3768.802839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.823320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3768.832148] Lustre: Skipped 2 previous similar messages [ 3771.342913] Lustre: server umount lustre-MDT0000 complete [ 3773.925275] LustreError: 127346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3773.947544] LustreError: 127346:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3775.248552] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865882 with bad export cookie 13606926104108311037 [ 3775.267104] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.282898] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3775.644410] Lustre: server umount lustre-MDT0001 complete [ 3787.099594] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 3789.829743] Key type lgssc unregistered [ 3790.127653] LNet: 130943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3790.136794] LNetError: 130943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3790.150928] LNet: Removed LNI 192.168.203.144@tcp [ 3791.089761] Key type .llcrypt unregistered [ 3791.092505] Key type ._llcrypt unregistered [ 3818.312929] Key type ._llcrypt registered [ 3818.317925] Key type .llcrypt registered [ 3818.425675] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3833.376713] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3834.234078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3834.256434] alg: No test for adler32 (adler32-zlib) [ 3835.362843] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 3835.564630] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 3837.281718] Key type lgssc registered [ 3838.401106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3873.612683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3873.627920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.871271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3874.895979] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3874.986337] Lustre: lustre-MDT0000: new disk, initializing [ 3875.069184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.084697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3880.545735] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3892.277452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.470949] Lustre: 134343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3892.617990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3892.623136] Lustre: Skipped 1 previous similar message [ 3892.726426] Lustre: lustre-MDT0001: new disk, initializing [ 3892.868253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3892.916561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3892.929512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3897.759108] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3908.634970] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3915.632165] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3922.389835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3922.645043] Lustre: lustre-OST0000: new disk, initializing [ 3922.648571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3922.655839] Lustre: 136310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3922.762623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.992922] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3931.661032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3931.669557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3931.775829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3937.825279] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.588437] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3941.790722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3946.331269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3946.483617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3952.103501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.105342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3952.135529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.411337] Lustre: server umount lustre-OST0000 complete [ 3959.265435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.286930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.312334] Lustre: Skipped 1 previous similar message [ 3959.317039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.335381] Lustre: Skipped 1 previous similar message [ 3962.338335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.338916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.354314] Lustre: Skipped 1 previous similar message [ 3967.460482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3967.476931] Lustre: Skipped 1 previous similar message [ 3972.585595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3972.595273] Lustre: Skipped 1 previous similar message [ 3973.087568] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3973.345567] Lustre: server umount lustre-MDT0000 complete [ 3976.999765] LustreError: 134337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866083 with bad export cookie 15621600568498482264 [ 3977.004548] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3977.010908] LustreError: 134337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3977.276971] Lustre: server umount lustre-MDT0001 complete [ 3985.447299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.993052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3990.577975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3991.010956] LustreError: 138531:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.044070] LustreError: 138531:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3996.129919] LustreError: 138530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.878624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.242415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.935655] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4010.082805] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4016.029108] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4022.910240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.240957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.570901] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4036.477981] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4040.854463] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4041.030234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4045.637407] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4045.954691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4056.282076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.544214] Lustre: lustre-OST0001: new disk, initializing [ 4056.551326] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4056.565505] Lustre: 141903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4056.681347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4063.268434] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4065.863931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4065.884184] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4066.036209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4071.268166] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4071.522047] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4076.303747] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4076.529454] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4082.358445] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9325c6d72800.ost_server_uuid 50 [ 4083.924030] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9325c6d72800.ost_server_uuid in IDLE state after 0 sec [ 4096.994153] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4097.020332] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.055240] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4097.065873] Lustre: Skipped 1 previous similar message [ 4107.233413] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4107.400623] Lustre: server umount lustre-OST0001 complete [ 4112.355846] LustreError: 140352:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.357518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.378721] LustreError: 140352:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4112.405423] Lustre: Skipped 2 previous similar messages [ 4117.474380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.485140] Lustre: Skipped 7 previous similar messages [ 4117.490236] LustreError: 140352:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.520441] LustreError: 140352:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4117.948855] Lustre: server umount lustre-OST0000 complete [ 4122.594330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.201101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4127.211523] LustreError: Skipped 1 previous similar message [ 4128.685565] Lustre: server umount lustre-MDT0000 complete [ 4132.835514] LustreError: 139258:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.860029] LustreError: 139258:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4132.970623] LustreError: 138511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866239 with bad export cookie 15621600568498483146 [ 4132.973093] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4132.994047] LustreError: 138511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4133.504714] Lustre: server umount lustre-MDT0001 complete [ 4155.343637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 4158.708289] Key type lgssc unregistered [ 4159.128815] LNet: 144921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4159.149500] LNetError: 144921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4159.185104] LNet: Removed LNI 192.168.203.144@tcp [ 4160.083869] Key type .llcrypt unregistered [ 4160.086080] Key type ._llcrypt unregistered [ 4204.624481] Key type ._llcrypt registered [ 4204.628372] Key type .llcrypt registered [ 4204.775133] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4220.982374] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4222.248252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4222.309376] alg: No test for adler32 (adler32-zlib) [ 4223.455680] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 4223.734103] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4225.431213] Key type lgssc registered [ 4226.761911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4263.628493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4263.668047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.941175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4264.992848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4265.090923] Lustre: lustre-MDT0000: new disk, initializing [ 4265.183461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.233546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4270.023504] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4282.052810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.195583] Lustre: 149079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.242844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4282.246587] Lustre: Skipped 1 previous similar message [ 4282.302282] Lustre: lustre-MDT0001: new disk, initializing [ 4282.379831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.419321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4282.435560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4287.125892] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4296.898095] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4302.478626] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4309.072566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.371888] Lustre: lustre-OST0000: new disk, initializing [ 4309.375539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4309.383521] Lustre: 151046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4309.480136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.609949] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4319.279278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4319.301276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4319.410328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4325.202088] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4329.172992] Lustre: Failing over lustre-OST0000 [ 4329.371245] Lustre: server umount lustre-OST0000 complete [ 4345.311141] Lustre: 147301:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866436/real 1787866436] req@0000000098439541 x1874713724879104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787866452 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4345.316377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.339338] Lustre: 147301:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4345.357476] Lustre: Skipped 1 previous similar message [ 4347.363220] Lustre: 147302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866438/real 1787866438] req@00000000fba5929b x1874713724879488/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787866454 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4347.390396] Lustre: 147302:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4348.303982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4348.764917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.781803] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4350.432573] Lustre: 147303:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866441/real 1787866441] req@0000000051c1a488 x1874713724880384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787866457 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4353.513889] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4355.246793] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4356.000309] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4356.008448] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4357.023855] Lustre: 147300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866447/real 1787866447] req@00000000bc7460a4 x1874713724881664/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787866463 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4357.065598] Lustre: 147300:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4359.133866] Lustre: Failing over lustre-MDT0000 [ 4359.141048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 4359.149932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.172488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4359.383222] Lustre: server umount lustre-MDT0000 complete [ 4361.189096] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.192297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.208362] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4362.271135] Lustre: 147300:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866452/real 1787866452] req@00000000c13df291 x1874713724882688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787866468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4362.304392] Lustre: 147300:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4363.790526] LustreError: 150844:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.802031] LustreError: 150844:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4366.310276] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.330200] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4368.864254] LustreError: 149086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.009138] LustreError: 149085:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.032692] LustreError: 149085:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4377.570417] Lustre: 147302:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787866468/real 1787866468] req@00000000f70055d3 x1874713724892032/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787866484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4377.604320] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4378.440272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.688990] LustreError: 150844:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.709281] LustreError: 150844:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 4388.158611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.237866] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4389.360697] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4393.487417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4393.504019] Lustre: Skipped 1 previous similar message [ 4393.527427] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4393.591972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4394.086309] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4402.146549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.154088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.171756] Lustre: Skipped 1 previous similar message [ 4402.176974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4403.694391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4403.706954] Lustre: Skipped 1 previous similar message [ 4408.236688] Lustre: server umount lustre-OST0000 complete [ 4413.920487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4413.931318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.942576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.951459] Lustre: Skipped 2 previous similar messages [ 4413.971886] Lustre: Skipped 1 previous similar message [ 4418.723455] Lustre: server umount lustre-MDT0000 complete [ 4419.046789] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.062221] LustreError: 149089:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4422.565416] LustreError: 149071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866529 with bad export cookie 10121847617283184200 [ 4422.566536] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.577936] LustreError: 149071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4423.073799] Lustre: server umount lustre-MDT0001 complete [ 4434.799676] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 4437.735347] Key type lgssc unregistered [ 4438.071974] LNet: 154705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4438.083714] LNetError: 154705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4438.115034] LNet: Removed LNI 192.168.203.144@tcp [ 4439.099697] Key type .llcrypt unregistered [ 4439.102112] Key type ._llcrypt unregistered [ 4477.163183] Key type ._llcrypt registered [ 4477.164648] Key type .llcrypt registered [ 4477.303236] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4478.366926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4478.431131] alg: No test for adler32 (adler32-zlib) [ 4479.642828] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 4479.899366] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 4481.559169] Key type lgssc registered [ 4483.094823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4491.501554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4491.547563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.127949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.925342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4502.502103] LustreError: 156224:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.533915] LustreError: 156224:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4507.451950] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4507.635388] LustreError: 156225:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4511.394044] LustreError: 156253:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4511.406747] LustreError: 156253:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4511.419280] LustreError: 156253:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4511.871894] Lustre: server umount lustre-MDT0000 complete [ 4527.836374] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4537.560695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4573.882507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.238718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4574.265388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4574.329623] Lustre: lustre-MDT0000: new disk, initializing [ 4574.420290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4574.449854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4579.021842] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4588.752637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.840248] Lustre: 159559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4588.897948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4588.903315] Lustre: Skipped 1 previous similar message [ 4588.966694] Lustre: lustre-MDT0001: new disk, initializing [ 4589.078370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4589.127304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4589.138807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4593.903800] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4606.137356] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4613.511770] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4621.515118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4621.715329] Lustre: lustre-OST0000: new disk, initializing [ 4621.720374] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4621.738769] Lustre: 161531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4621.873677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.726812] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4628.536893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4628.553680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4628.642957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4637.516376] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4642.385421] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4644.297721] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:39:08 (1787866748) [ 4901.343977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.352919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.376067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.927948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.931943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4904.955145] Lustre: Skipped 1 previous similar message [ 4904.971927] Lustre: Skipped 2 previous similar messages [ 4909.334301] Lustre: server umount lustre-MDT0000 complete [ 4910.048378] LustreError: 159566:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.062216] LustreError: 159566:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4912.634429] LustreError: 160354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867019 with bad export cookie 11027770412571208811 [ 4912.635484] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.650663] LustreError: 160354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4915.168784] LustreError: 159567:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.173958] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.210549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4917.215780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4918.931405] Lustre: server umount lustre-MDT0001 complete [ 4928.557535] Lustre: server umount lustre-OST0000 complete [ 4939.522560] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4951.216878] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4989.377234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.529310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4989.549220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4989.603755] Lustre: lustre-MDT0000: new disk, initializing [ 4989.658972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.667986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4994.731408] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5005.553718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.728612] Lustre: 166193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5005.779505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5005.783796] Lustre: Skipped 1 previous similar message [ 5005.871814] Lustre: lustre-MDT0001: new disk, initializing [ 5005.976158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5006.030716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5006.045230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.936829] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5020.726992] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5026.367335] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5033.356043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5033.590849] Lustre: lustre-OST0000: new disk, initializing [ 5033.598242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5033.613241] Lustre: 168160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5033.719922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5040.066038] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5040.230749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5040.238365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5040.390306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5049.150841] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5054.010818] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5054.252928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5058.836773] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5059.095948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5061.088482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5061.094396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5061.097091] LustreError: Skipped 1 previous similar message [ 5061.120917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5061.132684] Lustre: Skipped 1 previous similar message [ 5066.971575] Lustre: server umount lustre-OST0000 complete [ 5071.329660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.351125] Lustre: Skipped 1 previous similar message [ 5071.358311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5071.364490] Lustre: Skipped 3 previous similar messages [ 5085.152920] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5085.370187] Lustre: server umount lustre-MDT0000 complete [ 5086.690148] LustreError: 167560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5086.705480] LustreError: 167560:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5089.066691] LustreError: 168159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867195 with bad export cookie 11027770412571367641 [ 5089.068778] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5089.080333] LustreError: 168159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5089.366610] Lustre: server umount lustre-MDT0001 complete [ 5097.497101] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:46:42 (1787867202) [ 5113.786388] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5123.903881] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5162.355582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.518535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5162.567110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5162.677555] Lustre: lustre-MDT0000: new disk, initializing [ 5162.780280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.814730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5167.239265] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5178.247244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5178.417250] Lustre: 172916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5178.697791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5182.839031] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5192.156419] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.366991] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5205.648498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.889510] Lustre: lustre-OST0000: new disk, initializing [ 5205.892387] Lustre: Skipped 1 previous similar message [ 5205.895764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5205.899815] Lustre: Skipped 2 previous similar messages [ 5205.902947] Lustre: 174893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5212.309679] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5213.303544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5213.310041] Lustre: Skipped 1 previous similar message [ 5213.315047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5213.424472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5222.246313] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5234.144891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5234.154293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.161293] LustreError: Skipped 1 previous similar message [ 5234.162536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.175355] Lustre: Skipped 2 previous similar messages [ 5234.181634] Lustre: Skipped 4 previous similar messages [ 5235.527319] Lustre: server umount lustre-OST0000 complete [ 5245.426367] Lustre: server umount lustre-MDT0000 complete [ 5249.048104] LustreError: 174892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867355 with bad export cookie 11027770412571368523 [ 5249.056212] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5249.059120] LustreError: 174892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5249.427647] Lustre: server umount lustre-MDT0001 complete [ 5257.866074] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:49:22 (1787867362) [ 5274.271838] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5285.386822] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5319.990915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.177842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5320.251563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5320.356795] Lustre: lustre-MDT0000: new disk, initializing [ 5320.458884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.465344] Lustre: Skipped 2 previous similar messages [ 5320.483113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5325.979777] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5338.321079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.528625] Lustre: 179221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5338.881334] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5344.115952] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5355.268387] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5361.493797] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5368.835859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5369.196424] Lustre: 181198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5374.745683] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5377.592247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5377.741187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5383.956503] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5393.387133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5393.400443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5393.404290] Lustre: Skipped 2 previous similar messages [ 5393.409257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5393.438024] Lustre: Skipped 6 previous similar messages [ 5398.137340] Lustre: server umount lustre-OST0000 complete [ 5408.875917] Lustre: server umount lustre-MDT0000 complete [ 5412.567574] LustreError: 179213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867519 with bad export cookie 11027770412571369608 [ 5412.569124] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.577179] LustreError: 179213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5413.040400] Lustre: server umount lustre-MDT0001 complete [ 5424.398398] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5427.648449] Key type lgssc unregistered [ 5427.948884] LNet: 183057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5427.958679] LNetError: 183057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5427.976526] LNet: Removed LNI 192.168.203.144@tcp [ 5428.756598] Key type .llcrypt unregistered [ 5428.758264] Key type ._llcrypt unregistered [ 5452.939190] Key type ._llcrypt registered [ 5452.941202] Key type .llcrypt registered [ 5453.059224] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5454.096041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5454.138671] alg: No test for adler32 (adler32-zlib) [ 5455.166465] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 5455.371965] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5457.023357] Key type lgssc registered [ 5458.142820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5465.489471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5465.534614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.023061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.806842] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5476.323950] LustreError: 184473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.344638] LustreError: 184473:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5479.833351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.167668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5484.530557] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5493.042755] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5498.862253] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5508.373504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5509.147217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.251608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5519.304419] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5528.230356] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5550.056625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.059632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.073682] Lustre: Skipped 1 previous similar message [ 5550.091631] Lustre: Skipped 1 previous similar message [ 5553.619417] Lustre: server umount lustre-OST0000 complete [ 5560.289235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.301125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.311377] Lustre: Skipped 1 previous similar message [ 5560.321526] Lustre: Skipped 1 previous similar message [ 5563.862900] Lustre: server umount lustre-MDT0000 complete [ 5565.420060] LustreError: 185200:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5565.472688] LustreError: 185200:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5567.891184] LustreError: 184452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867674 with bad export cookie 10627250542943568153 [ 5567.900260] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.276073] Lustre: server umount lustre-MDT0001 complete [ 5579.360417] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5581.998800] Key type lgssc unregistered [ 5582.370878] LNet: 188034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5582.377647] LNetError: 188034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5582.415862] LNet: Removed LNI 192.168.203.144@tcp [ 5583.410199] Key type .llcrypt unregistered [ 5583.416891] Key type ._llcrypt unregistered [ 5610.518948] Key type ._llcrypt registered [ 5610.527990] Key type .llcrypt registered [ 5610.681937] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5612.061858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5612.102152] alg: No test for adler32 (adler32-zlib) [ 5613.274487] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 5613.640034] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 5615.383235] Key type lgssc registered [ 5616.604759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5624.722598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5624.744889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.137277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.501694] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5635.555234] LustreError: 189454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5635.569590] LustreError: 189454:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5640.130851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.457129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5644.845612] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5652.102589] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5657.718968] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5664.865883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.241650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5670.407111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5672.074251] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5679.876385] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5685.730773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5685.743283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.748312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5685.760745] Lustre: Skipped 1 previous similar message [ 5686.785916] Lustre: lustre-OST0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 5686.800410] Lustre: Skipped 1 previous similar message [ 5690.853511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5691.469168] Lustre: server umount lustre-OST0000 complete [ 5699.243054] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9325c774d800.ost_server_uuid 50 [ 5701.023843] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5715.941264] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.947283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.965432] Lustre: Skipped 1 previous similar message [ 5715.974793] Lustre: Skipped 1 previous similar message [ 5721.059863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5721.067794] Lustre: Skipped 1 previous similar message [ 5721.109632] Lustre: server umount lustre-MDT0000 complete [ 5724.935625] LustreError: 189434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867831 with bad export cookie 13266061880439884654 [ 5724.939065] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5724.947411] LustreError: 189434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5725.189967] Lustre: server umount lustre-MDT0001 complete [ 5734.255700] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:57:18 (1787867838) [ 5741.300826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5742.133636] LustreError: 193367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.206561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.895979] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5747.682471] LustreError: 193368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.782903] LustreError: 193367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.345374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.674897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5759.651789] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5767.543721] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5773.627904] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5781.727805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5782.116695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.245823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5788.381125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5796.025537] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5802.780576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5803.025612] Lustre: lustre-OST0001: new disk, initializing [ 5803.037491] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5803.052238] Lustre: 196210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5803.187681] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5809.535166] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5812.843236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5812.859931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5812.978444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5818.530906] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5823.459916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5823.470807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.477203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5823.479769] Lustre: Skipped 1 previous similar message [ 5827.719212] Lustre: server umount lustre-OST0000 complete [ 5828.592741] LustreError: 196068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.618566] LustreError: 196068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5833.923453] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5834.173795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5838.260727] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5838.490039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5838.823620] LustreError: 196068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.845728] LustreError: 196068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5854.186977] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.202814] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.216106] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5854.233536] Lustre: Skipped 3 previous similar messages [ 5858.506816] Lustre: server umount lustre-OST0001 complete [ 5863.394768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5863.404834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.418497] Lustre: Skipped 1 previous similar message [ 5869.075315] Lustre: server umount lustre-MDT0000 complete [ 5869.537446] LustreError: 193367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5869.568609] LustreError: 193367:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5872.994126] LustreError: 193347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867979 with bad export cookie 13266061880439885725 [ 5872.997401] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.009727] LustreError: 193347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5873.059694] LustreError: 188765:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 5873.474989] Lustre: server umount lustre-MDT0001 complete [ 5899.618733] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:00:04 (1787868004) [ 5906.999567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.436808] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5907.502996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5907.655138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.594248] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5921.966039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.125240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5922.171884] Lustre: 199646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5927.679863] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5935.818154] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5941.453720] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5948.502791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.655214] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5955.715392] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5958.292904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5963.596718] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.287218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5970.349804] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5976.888268] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 5980.924827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5985.971479] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5989.346295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5989.351678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.362700] Lustre: Skipped 1 previous similar message [ 5989.366862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.380283] Lustre: Skipped 3 previous similar messages [ 5995.533407] Lustre: server umount lustre-OST0000 complete [ 5996.536506] LustreError: 201496:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5996.557616] LustreError: 201496:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6017.013174] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6017.027182] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6017.047316] Lustre: Skipped 1 previous similar message [ 6017.068076] LustreError: Skipped 1 previous similar message [ 6020.244961] Lustre: server umount lustre-OST0001 complete [ 6030.661340] Lustre: server umount lustre-MDT0000 complete [ 6034.063114] LustreError: 202544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868140 with bad export cookie 13266061880439887132 [ 6034.068175] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6034.074323] LustreError: 202544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6034.358550] Lustre: server umount lustre-MDT0001 complete [ 6058.611689] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:02:43 (1787868163) [ 6075.837578] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6087.916332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6128.485833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.719367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6128.748266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6128.828213] Lustre: lustre-MDT0000: new disk, initializing [ 6128.902507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6128.906710] Lustre: Skipped 3 previous similar messages [ 6128.922554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6133.479973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6144.341966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.474529] Lustre: 207755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6144.538594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6144.543595] Lustre: Skipped 1 previous similar message [ 6144.630429] Lustre: lustre-MDT0001: new disk, initializing [ 6144.841786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6144.857190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6149.775707] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6160.775806] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6168.186822] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6177.009830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6177.409636] Lustre: lustre-OST0000: new disk, initializing [ 6177.419938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6177.430768] Lustre: 209727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6184.017057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6184.021415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6184.126554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6184.297713] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6194.108895] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6199.362492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6199.700670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6204.751623] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6204.958773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6210.016348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.017450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.025204] LustreError: Skipped 2 previous similar messages [ 6210.026390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6210.026443] Lustre: Skipped 8 previous similar messages [ 6210.048320] Lustre: Skipped 4 previous similar messages [ 6221.279127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6221.477299] Lustre: server umount lustre-OST0000 complete [ 6239.199130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6239.439180] Lustre: server umount lustre-MDT0000 complete [ 6240.743114] LustreError: 207768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.772508] LustreError: 207768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 6242.851760] LustreError: 207747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868349 with bad export cookie 13266061880439888917 [ 6242.859193] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.863681] LustreError: 207747:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6243.283148] Lustre: server umount lustre-MDT0001 complete [ 6251.288425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.993226] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6264.122880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.475848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6264.487696] Lustre: Skipped 3 previous similar messages [ 6268.395568] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6275.392652] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6280.979641] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6289.045238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6296.314265] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6304.348982] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6308.899225] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6309.071695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6313.782981] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6313.988324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6317.030055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.033598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6317.048426] Lustre: Skipped 3 previous similar messages [ 6330.335807] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6330.492391] Lustre: server umount lustre-OST0000 complete [ 6337.173766] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6341.835092] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6370.150356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6375.498283] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6382.707158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.741898] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.754591] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6382.785032] Lustre: Skipped 1 previous similar message [ 6383.562173] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6390.116947] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6398.442156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6398.456032] LustreError: Skipped 1 previous similar message [ 6398.460265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6398.472781] Lustre: Skipped 18 previous similar messages [ 6400.690608] Lustre: server umount lustre-OST0000 complete [ 6412.279729] Lustre: server umount lustre-MDT0000 complete [ 6413.797545] LustreError: 216417:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.843211] LustreError: 216417:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6415.617119] LustreError: 211926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868522 with bad export cookie 13266061880439889799 [ 6415.619452] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.639495] LustreError: 211926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6415.879557] Lustre: server umount lustre-MDT0001 complete [ 6426.584416] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:08:51 (1787868531) [ 6434.172171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6439.790414] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6448.550381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.881101] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6461.261230] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6467.135265] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6473.908466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6480.928383] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6488.445598] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6493.350038] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6493.564453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6498.172502] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6498.405767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6505.395285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.644201] Lustre: lustre-OST0001: new disk, initializing [ 6505.646785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6505.665583] Lustre: 221287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.622032] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6513.273305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6513.290283] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6513.441679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6520.701758] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.374210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6525.564206] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6530.056490] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6530.263816] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6534.112797] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.129405] Lustre: Skipped 5 previous similar messages [ 6538.529512] Lustre: server umount lustre-OST0001 complete [ 6545.015232] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6545.245329] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6550.275908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6550.461943] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6579.291928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.526039] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.536890] Lustre: Skipped 6 previous similar messages [ 6584.823314] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6584.833840] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6584.840101] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6587.370391] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6596.784657] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.971716] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6605.281609] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6605.308531] LustreError: Skipped 2 previous similar messages [ 6607.250709] Lustre: server umount lustre-OST0001 complete [ 6629.519871] Lustre: server umount lustre-OST0000 complete [ 6639.859446] Lustre: server umount lustre-MDT0000 complete [ 6643.420376] LustreError: 217898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868750 with bad export cookie 13266061880439890765 [ 6643.425621] LustreError: 217898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6643.430636] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6649.832891] Lustre: server umount lustre-MDT0001 complete [ 6673.466228] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:12:58 (1787868778) [ 6679.744713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.026430] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6680.108837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6684.680652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6692.809915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6692.907547] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6692.943209] Lustre: 226528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6698.289234] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6706.189658] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.280875] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6718.233312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6718.454195] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6724.513425] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6731.800866] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6741.330844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.411660] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6747.672807] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6755.603385] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.322945] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6762.840839] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6763.099939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6767.874302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6768.110396] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6774.385918] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9325c44cd000.ost_server_uuid 50 [ 6775.936639] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9325c44cd000.ost_server_uuid in IDLE state after 0 sec [ 6776.932920] Lustre: Permanently deactivating lustre-OST0001 [ 6776.945478] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6781.748706] Lustre: Permanently reactivating lustre-OST0001 [ 6781.776452] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6781.782521] Lustre: Skipped 2 previous similar messages [ 6788.067731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6788.074447] Lustre: Skipped 11 previous similar messages [ 6792.860255] Lustre: server umount lustre-OST0001 complete [ 6794.214878] LustreError: 228373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.232613] LustreError: 228373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 36 previous similar messages [ 6798.816612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.840798] Lustre: Skipped 8 previous similar messages [ 6802.255465] Lustre: server umount lustre-OST0000 complete [ 6811.498976] Lustre: server umount lustre-MDT0000 complete [ 6815.154378] LustreError: 226521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868922 with bad export cookie 13266061880439892053 [ 6815.155215] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.170067] LustreError: 226521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6815.571367] Lustre: server umount lustre-MDT0001 complete [ 6837.806907] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:15:42 (1787868942) [ 6846.393927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.889577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6846.937862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6846.946958] Lustre: Skipped 2 previous similar messages [ 6851.931317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6860.239276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.302289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6860.323211] Lustre: 233226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6864.646385] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6872.223611] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6878.094351] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6885.611701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6885.791150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6885.805085] Lustre: Permanently deactivating lustre-OST0000 [ 6885.815836] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6893.618938] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6894.209729] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6903.780955] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6910.657572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6910.811630] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6918.881332] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6921.419583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6927.084678] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6928.842599] Lustre: Permanently reactivating lustre-OST0000 [ 6928.885513] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6928.889818] Lustre: Skipped 2 previous similar messages [ 6934.613875] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6934.613933] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 0000000000f41be4, cur 1787869041 deadline 1787869030 last 1787869000 [ 6934.623991] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6934.624327] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6934.624331] Lustre: Skipped 1 previous similar message [ 6934.696823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6945.284947] Lustre: server umount lustre-OST0001 complete [ 6947.304760] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6947.310898] LustreError: Skipped 6 previous similar messages [ 6954.739822] Lustre: server umount lustre-OST0000 complete [ 6958.020658] Lustre: server umount lustre-MDT0000 complete [ 6961.429494] LustreError: 233219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869068 with bad export cookie 13266061880439893726 [ 6961.442601] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.447655] LustreError: 233219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6961.776761] Lustre: server umount lustre-MDT0001 complete [ 6971.740181] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6973.832703] Key type lgssc unregistered [ 6974.042500] LNet: 238106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6974.047288] LNetError: 238106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6974.067133] LNet: Removed LNI 192.168.203.144@tcp [ 6974.742848] Key type .llcrypt unregistered [ 6974.747648] Key type ._llcrypt unregistered [ 6995.624629] Key type ._llcrypt registered [ 6995.628227] Key type .llcrypt registered [ 6995.756177] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6997.012328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6997.115223] alg: No test for adler32 (adler32-zlib) [ 6998.154382] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 6998.395093] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7000.103544] Key type lgssc registered [ 7001.123584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7011.255871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7011.288705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.847606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.331690] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7023.078596] LustreError: 239575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.092370] LustreError: 239575:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7024.974262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.114062] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7025.141590] Lustre: 239563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7025.159962] Lustre: Permanently deactivating lustre-MDT0001 [ 7025.184714] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7025.559172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7026.257812] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7026.263931] LustreError: 239606:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7026.283538] LustreError: 239606:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7026.296132] LustreError: 239606:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7029.798410] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7036.721707] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7041.996731] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7048.651190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.107685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.184771] LustreError: 241418:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7051.202822] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7055.648644] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7056.356496] LustreError: 241547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.417350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7062.523570] LustreError: 241419:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.553735] LustreError: 241419:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7064.669323] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7067.625571] LustreError: 241547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.946621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7073.179748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7078.391584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7078.400193] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7079.526242] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7087.277909] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7090.883184] Lustre: Permanently reactivating lustre-MDT0001 [ 7090.909918] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7090.921897] Lustre: Skipped 2 previous similar messages [ 7095.862727] Lustre: lustre-MDT0001: Client 77b4c3c7-b955-4754-b61a-d86fad175583 (at 192.168.203.44@tcp) reconnecting [ 7103.462721] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.541903] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7111.847955] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7112.098317] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7117.312771] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7117.518546] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7120.535616] Lustre: Permanently deactivating lustre-MDT0001 [ 7120.561414] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7120.568178] Lustre: Skipped 2 previous similar messages [ 7126.161430] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7135.714816] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7135.716847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7135.756685] Lustre: Skipped 1 previous similar message [ 7135.779678] Lustre: Skipped 1 previous similar message [ 7137.192622] Lustre: server umount lustre-MDT0000 complete [ 7140.841946] LustreError: 239575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.871276] LustreError: 239575:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 7141.371929] LustreError: 239555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869248 with bad export cookie 9203073946063804583 [ 7141.376617] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.397937] LustreError: 239555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7141.924865] Lustre: server umount lustre-MDT0001 complete [ 7147.930654] Lustre: server umount lustre-OST0000 complete [ 7160.269667] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:21:04 (1787869264) [ 7170.968070] Lustre: server umount lustre-OST0001 complete [ 7178.775381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7191.248083] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7229.955611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7230.294212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7230.305291] Lustre: Skipped 2 previous similar messages [ 7230.348778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7230.476891] Lustre: lustre-MDT0000: new disk, initializing [ 7230.579254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7230.604752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7235.466040] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7245.674865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7245.807684] Lustre: 247835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7245.867383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7245.875052] Lustre: Skipped 1 previous similar message [ 7245.959823] Lustre: lustre-MDT0001: new disk, initializing [ 7246.092670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7246.151235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7246.184763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7250.397571] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7258.844320] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7263.940989] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.653636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.994168] Lustre: lustre-OST0000: new disk, initializing [ 7270.997242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7271.006215] Lustre: 249800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7271.137323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.603698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7276.623185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7276.724133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7277.129210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7287.112491] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7289.197691] LustreError: 250333:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7298.162928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7298.361798] Lustre: lustre-OST0001: new disk, initializing [ 7298.366960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7298.374632] Lustre: 250915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7299.199079] LustreError: 250333:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7299.227333] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7305.161501] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7306.929697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7306.949326] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7307.097766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7309.280239] LustreError: 247840:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7316.138942] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7318.227100] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7322.593345] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7322.594493] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.608322] LustreError: Skipped 1 previous similar message [ 7322.623707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7322.627196] Lustre: Skipped 3 previous similar messages [ 7322.653404] Lustre: Skipped 1 previous similar message [ 7326.561025] Lustre: server umount lustre-OST0001 complete [ 7327.719575] LustreError: 251363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7327.745634] LustreError: 251363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7343.584932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7343.589396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.590657] LustreError: Skipped 1 previous similar message [ 7343.593329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.619180] Lustre: Skipped 1 previous similar message [ 7348.193134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7348.207393] Lustre: Skipped 2 previous similar messages [ 7349.763420] Lustre: server umount lustre-OST0000 complete [ 7353.314724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.320796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7353.348351] Lustre: Skipped 1 previous similar message [ 7353.360653] Lustre: Skipped 1 previous similar message [ 7359.549468] Lustre: server umount lustre-MDT0000 complete [ 7363.199146] LustreError: 248628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869470 with bad export cookie 9203073946063807621 [ 7363.205720] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7363.213167] LustreError: 248628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7369.587488] Lustre: server umount lustre-MDT0001 complete [ 7380.890528] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7383.342122] Key type lgssc unregistered [ 7383.665911] LNet: 252957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7383.673394] LNetError: 252957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7383.688143] LNet: Removed LNI 192.168.203.144@tcp [ 7384.399364] Key type .llcrypt unregistered [ 7384.401688] Key type ._llcrypt unregistered [ 7421.854884] Key type ._llcrypt registered [ 7421.859346] Key type .llcrypt registered [ 7421.952752] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7423.134246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.198525] alg: No test for adler32 (adler32-zlib) [ 7424.221519] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7424.427207] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7426.135136] Key type lgssc registered [ 7427.272198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7434.183855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7434.198503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.471058] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7435.498947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7435.623244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.992772] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7450.582268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7450.721198] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7450.761469] Lustre: 255169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7451.085091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.196527] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7462.554324] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7468.161560] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7475.602416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.809111] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7476.061762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.269421] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7483.578940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7491.561185] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.806991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7544.822974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7544.845258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.329889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.342349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.360689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.372857] Lustre: Skipped 1 previous similar message [ 7550.214575] Lustre: server umount lustre-OST0000 complete [ 7554.625030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7568.798983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.153453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.189180] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7571.209299] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7575.871875] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7583.366581] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7585.584932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7585.600217] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7624.681512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.683758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.712492] Lustre: Skipped 1 previous similar message [ 7624.727601] Lustre: Skipped 1 previous similar message [ 7629.675718] Lustre: server umount lustre-OST0000 complete [ 7634.920929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.930558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7634.938430] Lustre: Skipped 1 previous similar message [ 7634.960664] Lustre: Skipped 1 previous similar message [ 7639.762179] Lustre: server umount lustre-MDT0000 complete [ 7643.524313] LustreError: 255161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869750 with bad export cookie 15733426424766013861 [ 7643.527955] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7643.542812] LustreError: 255161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7643.912865] Lustre: server umount lustre-MDT0001 complete [ 7654.934661] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7657.879687] Key type lgssc unregistered [ 7658.235631] LNet: 260595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7658.244573] LNetError: 260595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7658.259773] LNet: Removed LNI 192.168.203.144@tcp [ 7659.156934] Key type .llcrypt unregistered [ 7659.161294] Key type ._llcrypt unregistered [ 7684.977537] Key type ._llcrypt registered [ 7684.981632] Key type .llcrypt registered [ 7685.155309] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7686.507858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7686.549543] alg: No test for adler32 (adler32-zlib) [ 7687.670055] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7688.025894] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7689.743306] Key type lgssc registered [ 7690.895925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7698.388457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7698.415793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7699.820632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7704.305637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7709.158566] LustreError: 262011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.182612] LustreError: 262011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7712.648923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.996776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7716.993188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7725.057587] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7731.347540] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7738.238503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7738.599223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.757368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7745.137882] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7752.692823] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.952203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7785.957173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.987288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.029175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.034552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.051026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.281590] Lustre: server umount lustre-OST0000 complete [ 7799.278138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.289135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.313119] Lustre: Skipped 1 previous similar message [ 7799.320053] Lustre: Skipped 1 previous similar message [ 7802.258731] Lustre: server umount lustre-MDT0000 complete [ 7804.384502] LustreError: 262739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.401844] LustreError: 262739:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7806.374702] LustreError: 261992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869913 with bad export cookie 8035687089403941179 [ 7806.391478] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7806.394436] LustreError: 261992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7806.790243] Lustre: server umount lustre-MDT0001 complete [ 7818.201620] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7820.661128] Key type lgssc unregistered [ 7820.847133] LNet: 266323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.853156] LNetError: 266323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.891614] LNet: Removed LNI 192.168.203.144@tcp [ 7821.594492] Key type .llcrypt unregistered [ 7821.597291] Key type ._llcrypt unregistered [ 7844.904027] Key type ._llcrypt registered [ 7844.908912] Key type .llcrypt registered [ 7845.022152] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7846.646832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7846.699488] alg: No test for adler32 (adler32-zlib) [ 7847.768393] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7848.059724] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7849.817205] Key type lgssc registered [ 7851.108732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7859.116832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7859.158577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.743778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.490565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7870.953636] LustreError: 267547:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7870.987679] LustreError: 267547:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7873.388733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.750239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7878.518270] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7886.773340] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7892.920474] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7900.389298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7900.774812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7905.922961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7907.410820] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7915.160697] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7931.877578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7931.889286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7931.915170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.924335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.934160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.945031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.976937] Lustre: Skipped 1 previous similar message [ 7935.811084] Lustre: server umount lustre-OST0000 complete [ 7940.576828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7940.586418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7940.609865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7944.164146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7944.165383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7946.122248] Lustre: server umount lustre-MDT0000 complete [ 7949.287852] LustreError: 267542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.315612] LustreError: 267542:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7950.154245] LustreError: 267527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870057 with bad export cookie 17053117771376978692 [ 7950.157694] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7950.170521] LustreError: 267527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7950.457861] Lustre: server umount lustre-MDT0001 complete [ 7961.635917] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7964.383766] Key type lgssc unregistered [ 7964.714782] LNet: 271228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7964.729364] LNetError: 271228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7964.753620] LNet: Removed LNI 192.168.203.144@tcp [ 7965.521722] Key type .llcrypt unregistered [ 7965.524312] Key type ._llcrypt unregistered [ 7993.202463] Key type ._llcrypt registered [ 7993.224995] Key type .llcrypt registered [ 7993.388111] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7994.737930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7995.096563] alg: No test for adler32 (adler32-zlib) [ 7996.433215] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7996.838881] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7998.639168] Key type lgssc registered [ 7999.922878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8008.491431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8008.529978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.117263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8015.418863] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8019.431480] LustreError: 272645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.450531] LustreError: 272645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8024.547820] LustreError: 272646:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.633567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.100572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8030.129124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8038.376159] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.134440] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8051.050543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.360259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.456574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8057.983215] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8065.710678] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8095.716220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8095.725487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.751579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8099.816738] Lustre: server umount lustre-OST0000 complete [ 8105.959805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.973689] Lustre: Skipped 2 previous similar messages [ 8105.974609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.991558] Lustre: Skipped 2 previous similar messages [ 8109.641728] Lustre: server umount lustre-MDT0000 complete [ 8111.077960] LustreError: 273943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.115756] LustreError: 273943:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8113.666741] LustreError: 272627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870220 with bad export cookie 14724335945789894242 [ 8113.684394] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8113.694021] LustreError: 272627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8113.922539] Lustre: server umount lustre-MDT0001 complete [ 8125.519952] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8128.872246] Key type lgssc unregistered [ 8129.148912] LNet: 276958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8129.159093] LNetError: 276958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8129.179753] LNet: Removed LNI 192.168.203.144@tcp [ 8130.147508] Key type .llcrypt unregistered [ 8130.152109] Key type ._llcrypt unregistered [ 8149.471355] Key type ._llcrypt registered [ 8149.474234] Key type .llcrypt registered [ 8149.568793] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8150.520478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.616827] alg: No test for adler32 (adler32-zlib) [ 8151.849845] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 8152.059287] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8153.783264] Key type lgssc registered [ 8155.192278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.208824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.234768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.886933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.776354] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8175.081278] LustreError: 278199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.098734] LustreError: 278199:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8179.336628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.821761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8184.926967] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8194.036351] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8200.660620] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8208.752237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8209.034644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.102931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8214.897642] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8225.063643] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8240.098371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.120806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8240.151870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8240.192340] Lustre: Skipped 1 previous similar message [ 8245.230691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8245.233673] Lustre: Skipped 1 previous similar message [ 8245.783635] Lustre: server umount lustre-OST0000 complete [ 8250.342593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.359971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.362106] Lustre: Skipped 2 previous similar messages [ 8250.372182] Lustre: Skipped 1 previous similar message [ 8255.457111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.468761] Lustre: Skipped 1 previous similar message [ 8256.551512] Lustre: server umount lustre-MDT0000 complete [ 8260.576267] LustreError: 278182:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8260.599805] LustreError: 278182:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8260.644577] LustreError: 278164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870367 with bad export cookie 6030736907958110134 [ 8260.647792] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.662956] LustreError: 278164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8261.121383] Lustre: server umount lustre-MDT0001 complete [ 8272.742394] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8276.014758] Key type lgssc unregistered [ 8276.404772] LNet: 281905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8276.411200] LNetError: 281905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8276.435561] LNet: Removed LNI 192.168.203.144@tcp [ 8277.366908] Key type .llcrypt unregistered [ 8277.370049] Key type ._llcrypt unregistered [ 8289.495226] Key type ._llcrypt registered [ 8289.501171] Key type .llcrypt registered [ 8289.643576] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8306.893850] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8324.806183] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8326.655048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8326.690610] alg: No test for adler32 (adler32-zlib) [ 8327.738729] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 8328.096446] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8329.807268] Key type lgssc registered [ 8331.131152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8372.322415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8372.397472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.702131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8373.787920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8373.948599] Lustre: lustre-MDT0000: new disk, initializing [ 8374.096894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8374.150816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8380.483827] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8392.265403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.385340] Lustre: 285469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8392.501096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8392.515268] Lustre: Skipped 1 previous similar message [ 8392.602472] Lustre: lustre-MDT0001: new disk, initializing [ 8392.725151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8392.783163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8392.792113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8397.400695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8407.117217] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8413.482527] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8422.983754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8423.196830] Lustre: lustre-OST0000: new disk, initializing [ 8423.204861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8423.210369] Lustre: 287440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8423.305516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.121305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8432.228455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8432.250523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8432.486607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8441.450288] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.121889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8446.390927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8451.605678] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8451.847793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8458.209092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8458.219874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.229552] LustreError: Skipped 1 previous similar message [ 8458.248360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8458.256727] Lustre: Skipped 1 previous similar message [ 8460.085668] Lustre: server umount lustre-OST0000 complete [ 8464.352234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8464.363062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.379123] Lustre: Skipped 1 previous similar message [ 8464.384679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8468.451857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.453343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8468.475336] Lustre: Skipped 1 previous similar message [ 8470.431882] Lustre: server umount lustre-MDT0000 complete [ 8473.916649] LustreError: 287439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870580 with bad export cookie 1531164632416477421 [ 8473.927695] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8474.196804] Lustre: server umount lustre-MDT0001 complete [ 8482.309525] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:43:07 (1787870587) [ 8489.837795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.347679] LustreError: 289851:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.384607] LustreError: 289851:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8490.492546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.378704] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8495.585311] LustreError: 289852:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.708079] LustreError: 289851:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.057846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.532790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8511.052842] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8519.699442] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8526.573588] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8533.981114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8534.388256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.070270] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8546.958374] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8566.243579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8566.251078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.262132] LustreError: Skipped 1 previous similar message [ 8566.273086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8566.283311] Lustre: Skipped 1 previous similar message [ 8567.668897] Lustre: server umount lustre-OST0000 complete [ 8571.879055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8571.892809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.906994] Lustre: Skipped 1 previous similar message [ 8571.913142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.605846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.610867] Lustre: Skipped 3 previous similar messages [ 8586.207459] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8586.673843] Lustre: server umount lustre-MDT0000 complete [ 8586.726158] LustreError: 289846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8586.762384] LustreError: 289846:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8590.848290] LustreError: 289833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870697 with bad export cookie 1531164632416478310 [ 8590.856129] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8590.863456] LustreError: 289833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8591.346427] Lustre: server umount lustre-MDT0001 complete [ 8602.673316] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8606.155661] Key type lgssc unregistered [ 8606.443193] LNet: 293409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8606.449184] LNetError: 293409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8606.470424] LNet: Removed LNI 192.168.203.144@tcp [ 8607.279855] Key type .llcrypt unregistered [ 8607.282760] Key type ._llcrypt unregistered [ 8635.037734] Key type ._llcrypt registered [ 8635.042242] Key type .llcrypt registered [ 8635.163975] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8649.123536] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8650.303434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8650.462548] alg: No test for adler32 (adler32-zlib) [ 8651.684360] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 8651.989748] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8653.671171] Key type lgssc registered [ 8654.949534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8705.670867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8705.726325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.040192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8707.091461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8707.188688] Lustre: lustre-MDT0000: new disk, initializing [ 8707.302881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.329114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8712.236360] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8723.403018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.548542] Lustre: 297241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8723.620273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8723.628171] Lustre: Skipped 1 previous similar message [ 8723.713813] Lustre: lustre-MDT0001: new disk, initializing [ 8723.804749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.850505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8723.870674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8728.113600] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8738.553879] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8744.478317] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8751.929196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8752.235573] Lustre: lustre-OST2710: new disk, initializing [ 8752.248072] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8752.261433] Lustre: 299208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8752.400895] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8758.825312] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8758.845981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8758.853906] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8758.979435] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8768.024904] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8774.518419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8774.671427] Lustre: lustre-OST03e8: new disk, initializing [ 8774.674573] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8774.679837] Lustre: 300273:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8774.753608] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8781.311844] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8784.541767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8784.550136] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8784.697351] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8790.130232] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8798.111418] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8798.496915] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8803.930363] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8804.143752] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8810.014938] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8810.206067] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8815.454944] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8815.699455] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8820.705467] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.716237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8820.729937] Lustre: Skipped 2 previous similar messages [ 8820.760755] Lustre: Skipped 3 previous similar messages [ 8823.039803] Lustre: server umount lustre-MDT0000 complete [ 8826.933064] LustreError: 297230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870933 with bad export cookie 15454267060067024537 [ 8826.940682] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8826.945420] LustreError: 297230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8827.482606] Lustre: server umount lustre-MDT0001 complete [ 8842.086384] Lustre: server umount lustre-OST2710 complete [ 8856.697877] Lustre: server umount lustre-OST03e8 complete [ 8876.637664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8887.568442] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8925.975346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.203492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8926.224631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8926.294307] Lustre: lustre-MDT0000: new disk, initializing [ 8926.376321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.399930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8931.188922] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8941.817337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8941.971635] Lustre: 305523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8942.026559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8942.032119] Lustre: Skipped 1 previous similar message [ 8942.114791] Lustre: lustre-MDT0001: new disk, initializing [ 8942.233739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8942.272493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8942.289782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8947.541076] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8957.202313] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8963.223680] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8970.508709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.875504] Lustre: lustre-OST0000: new disk, initializing [ 8970.884521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8970.894303] Lustre: 307493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8970.997051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.372831] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8978.995230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8979.008214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8979.124348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8986.597255] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8990.855908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8991.083788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8995.179664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8995.457467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8999.929478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8999.934852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.938226] LustreError: Skipped 1 previous similar message [ 8999.939045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8999.958368] Lustre: Skipped 2 previous similar messages [ 9003.463316] Lustre: server umount lustre-OST0000 complete [ 9009.140691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9009.153201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.169737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9009.180899] Lustre: Skipped 1 previous similar message [ 9013.914138] Lustre: server umount lustre-MDT0000 complete [ 9015.264187] LustreError: 308914:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.282970] LustreError: 308914:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9017.620236] LustreError: 306327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871124 with bad export cookie 15454267060067026217 [ 9017.632063] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9017.637418] LustreError: 306327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9017.881363] Lustre: server umount lustre-MDT0001 complete [ 9026.424905] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:52:11 (1787871131) [ 9027.842608] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9029.624954] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:52:14 (1787871134) [ 9033.228464] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9059.240232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9059.624292] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9059.656491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9059.826450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.336434] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9072.254625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.340084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9072.356856] Lustre: 311350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9076.918979] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9099.446477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.745056] Lustre: Denying initial registration attempt from nid 192.168.203.144@tcp, specified as failover [ 9099.761630] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9099.780414] LustreError: 312806:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9099.800435] LustreError: 312806:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9099.815787] LustreError: 312806:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9099.924497] Lustre: server umount lustre-OST0000 complete [ 9099.931487] LustreError: 312806:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9106.915073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9106.924877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9106.934723] Lustre: Skipped 2 previous similar messages [ 9106.945903] Lustre: Skipped 2 previous similar messages [ 9112.033853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9112.041094] Lustre: Skipped 2 previous similar messages [ 9112.819768] Lustre: server umount lustre-MDT0000 complete [ 9117.162813] LustreError: 311363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.187959] LustreError: 311363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9117.243397] LustreError: 311344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871224 with bad export cookie 15454267060067027099 [ 9117.249816] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9117.259252] LustreError: 311344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9117.943810] Lustre: server umount lustre-MDT0001 complete [ 9130.433908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9133.602623] Key type lgssc unregistered [ 9134.027538] LNet: 313927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9134.035412] LNetError: 313927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9134.052512] LNet: Removed LNI 192.168.203.144@tcp [ 9134.917164] Key type .llcrypt unregistered [ 9134.922084] Key type ._llcrypt unregistered [ 9147.442907] Key type ._llcrypt registered [ 9147.444870] Key type .llcrypt registered [ 9147.589495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9148.395530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9148.421029] alg: No test for adler32 (adler32-zlib) [ 9149.558448] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9149.752808] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9151.415169] Key type lgssc registered [ 9152.736669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9198.821131] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9208.718959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9208.761729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9210.105807] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9210.137967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9210.370430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.060273] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9223.972928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9224.136716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9224.173504] Lustre: 316705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9224.505760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9229.192266] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9235.982241] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9241.599086] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9261.417043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9261.748120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9261.990959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9268.086612] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9276.209637] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9280.483296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9280.490927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9280.511747] LustreError: Skipped 1 previous similar message [ 9280.548737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9280.564679] Lustre: Skipped 1 previous similar message [ 9285.041131] Lustre: server umount lustre-OST0000 complete [ 9290.727596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9290.747819] Lustre: Skipped 1 previous similar message [ 9290.753671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9290.768516] Lustre: Skipped 1 previous similar message [ 9295.843888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.853146] Lustre: Skipped 1 previous similar message [ 9300.963266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.974797] Lustre: Skipped 1 previous similar message [ 9303.007180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9303.208646] Lustre: server umount lustre-MDT0000 complete [ 9306.695232] LustreError: 316697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871413 with bad export cookie 18441469826697374000 [ 9306.700757] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9306.712439] LustreError: 316697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9307.022915] Lustre: server umount lustre-MDT0001 complete [ 9317.859470] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9320.601791] Key type lgssc unregistered [ 9320.867724] LNet: 320283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9320.882392] LNetError: 320283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9320.897366] LNet: Removed LNI 192.168.203.144@tcp [ 9321.819806] Key type .llcrypt unregistered [ 9321.822989] Key type ._llcrypt unregistered [ 9347.086296] Key type ._llcrypt registered [ 9347.091603] Key type .llcrypt registered [ 9347.200412] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9348.554837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9348.590351] alg: No test for adler32 (adler32-zlib) [ 9349.694932] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9349.944209] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9351.647164] Key type lgssc registered [ 9352.612525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9360.480683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9360.509337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.899340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.527507] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9372.131152] LustreError: 321702:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9372.145944] LustreError: 321702:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9373.779866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9374.000101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9378.358178] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9386.292920] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9392.344271] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9400.167600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9400.542067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9406.173289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9412.498340] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9429.484914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.495615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.504206] Lustre: Skipped 2 previous similar messages [ 9429.513401] Lustre: Skipped 1 previous similar message [ 9431.041792] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 9434.269644] Lustre: server umount lustre-MDT0000 complete [ 9434.595019] LustreError: 321703:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9434.609723] LustreError: 321703:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9436.147816] LustreError: 321697:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.120044] LustreError: 321685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871544 with bad export cookie 1184103077364806759 [ 9438.126097] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9438.142458] LustreError: 321685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9439.715282] LustreError: 321698:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9439.715662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9439.744985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9443.746942] LustreError: 321698:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.747661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9443.758530] LustreError: 321698:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9443.772132] Lustre: Skipped 1 previous similar message [ 9444.490365] Lustre: server umount lustre-MDT0001 complete [ 9452.290255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9463.771467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.066673] LustreError: 323534:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9483.157154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9483.266215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9487.674528] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9495.758229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.184206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9497.197595] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9500.957235] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9508.709970] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9510.116369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9515.911743] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9517.766805] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9523.173774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.189892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9523.206607] Lustre: Skipped 1 previous similar message [ 9533.410296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9533.429254] Lustre: Skipped 4 previous similar messages [ 9536.991817] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9537.260447] Lustre: server umount lustre-OST0000 complete [ 9542.112510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9542.127064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9547.342536] Lustre: server umount lustre-MDT0000 complete [ 9548.775986] LustreError: 325324:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9548.798775] LustreError: 325324:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 9550.958460] LustreError: 321685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871657 with bad export cookie 1184103077364814305 [ 9550.968648] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9550.973650] LustreError: 321685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9551.323251] Lustre: server umount lustre-MDT0001 complete [ 9564.718440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9567.350836] Key type lgssc unregistered [ 9567.687457] LNet: 328009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9567.694108] LNetError: 328009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9567.708740] LNet: Removed LNI 192.168.203.144@tcp [ 9568.551626] Key type .llcrypt unregistered [ 9568.554407] Key type ._llcrypt unregistered [ 9593.532382] Key type ._llcrypt registered [ 9593.534627] Key type .llcrypt registered [ 9593.656637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9595.043791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9595.102935] alg: No test for adler32 (adler32-zlib) [ 9596.143528] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9596.362377] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9598.039218] Key type lgssc registered [ 9599.118475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9606.654205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.673098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9608.065245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9612.097753] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9617.379236] LustreError: 329426:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.408256] LustreError: 329426:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9622.440986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.497923] LustreError: 329427:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.051729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9629.916519] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9641.098814] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9646.415900] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9649.749729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.755625] Lustre: Skipped 1 previous similar message [ 9649.757222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9649.762992] Lustre: Skipped 1 previous similar message [ 9654.752384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9654.765994] Lustre: Skipped 1 previous similar message [ 9656.059574] Lustre: server umount lustre-MDT0000 complete [ 9659.316205] LustreError: 329406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871766 with bad export cookie 5275705110854002620 [ 9659.317798] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9659.324478] LustreError: 329406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9659.713473] Lustre: server umount lustre-MDT0001 complete [ 9676.055508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9676.507089] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9676.542631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9676.886196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.701501] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9692.465611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.654501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9692.720387] Lustre: 332020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9693.057728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9697.666879] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9704.920836] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9711.033730] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9722.105910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9722.370934] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9722.386109] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9722.400250] LustreError: 333907:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9722.414271] LustreError: 333907:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9722.420277] LustreError: 333907:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9722.556852] Lustre: server umount lustre-OST0000 complete [ 9722.563727] LustreError: 333907:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9733.669945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9733.847430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9733.995574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9739.852381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9742.489405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9748.665263] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9756.795949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9756.918790] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9757.008514] Lustre: lustre-OST0001: new disk, initializing [ 9757.016430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9757.024522] Lustre: 335449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9757.140211] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9762.932353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9762.952638] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9763.108730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9765.419355] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9774.591941] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9778.659988] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9778.671656] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9778.677510] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9778.680880] Lustre: Skipped 1 previous similar message [ 9783.781154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9784.243869] Lustre: server umount lustre-OST0001 complete [ 9788.388194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9788.394584] LustreError: Skipped 1 previous similar message [ 9788.397689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9788.411192] Lustre: Skipped 1 previous similar message [ 9788.419087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9788.425599] Lustre: Skipped 1 previous similar message [ 9788.900478] LustreError: 335711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.911896] LustreError: 335711:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9794.024179] LustreError: 334376:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.037107] LustreError: 334376:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9794.513837] Lustre: server umount lustre-OST0000 complete [ 9799.138861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9799.145097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9799.154695] Lustre: Skipped 2 previous similar messages [ 9799.169581] Lustre: Skipped 4 previous similar messages [ 9805.411390] Lustre: server umount lustre-MDT0000 complete [ 9809.385144] LustreError: 332031:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.404300] LustreError: 332031:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9810.271137] LustreError: 332770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871917 with bad export cookie 5275705110854003173 [ 9810.285827] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9810.888907] Lustre: server umount lustre-MDT0001 complete [ 9822.651364] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9825.444600] Key type lgssc unregistered [ 9825.750993] LNet: 337413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9825.758331] LNetError: 337413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9825.778488] LNet: Removed LNI 192.168.203.144@tcp [ 9826.755665] Key type .llcrypt unregistered [ 9826.766571] Key type ._llcrypt unregistered [ 9898.034681] Key type ._llcrypt registered [ 9898.037256] Key type .llcrypt registered [ 9898.226942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9914.244895] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9915.974866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9916.081718] alg: No test for adler32 (adler32-zlib) [ 9917.208525] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9917.459869] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9919.160085] Key type lgssc registered [ 9920.493673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9958.691026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9958.760644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.069421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9960.123932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9960.217482] Lustre: lustre-MDT0000: new disk, initializing [ 9960.328741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9960.340580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9964.957047] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9975.689876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.841836] Lustre: 342353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9975.915719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9975.920497] Lustre: Skipped 1 previous similar message [ 9975.988077] Lustre: lustre-MDT0001: new disk, initializing [ 9976.089396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.131875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9976.140826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9981.031063] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9994.601242] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10003.030376] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10010.827864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10011.307922] Lustre: lustre-OST0000: new disk, initializing [10011.314789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10011.330898] Lustre: 344324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10011.450803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10017.881466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10017.903074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10018.033428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10018.316804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10028.311433] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10033.572685] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10033.819484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10039.175303] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10039.482514] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10043.876720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10043.882062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.888176] LustreError: Skipped 1 previous similar message [10043.891344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10043.904667] Lustre: Skipped 1 previous similar message [10043.928642] Lustre: Skipped 1 previous similar message [10048.068141] Lustre: server umount lustre-OST0000 complete [10053.097320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10053.112982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10053.125491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.113559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10054.137540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.145372] Lustre: Skipped 1 previous similar message [10059.167744] Lustre: server umount lustre-MDT0000 complete [10063.428251] LustreError: 342343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872170 with bad export cookie 18146531607227915252 [10063.441198] LustreError: 342343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10063.447303] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.826217] Lustre: server umount lustre-MDT0001 complete [10073.948865] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:09:38 (1787872178) [10084.138574] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:09:48 (1787872188) [10094.462275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.162391] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.181715] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10095.239357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10100.335314] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10100.710303] LustreError: 347082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10105.829335] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.135624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.599131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10116.203162] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10125.507455] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10132.428285] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10141.128323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10141.467356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10148.684935] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10157.438631] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10175.813813] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10178.595942] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10181.388706] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10183.537927] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10186.812884] Lustre: Failing over lustre-MDT0000 [10187.250783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10187.261562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10187.282157] LustreError: 347082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.412210] Lustre: server umount lustre-MDT0000 complete [10191.842493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.843205] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.862073] Lustre: Skipped 1 previous similar message [10191.876392] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10202.097722] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10202.132569] LustreError: 347081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [10207.205685] Lustre: 340574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872298/real 1787872298] req@00000000ecb7e135 x1874719695515392/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787872314 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10207.261913] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10207.373030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10217.776264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10217.833462] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10220.528117] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10222.991244] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10223.103379] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10223.118319] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10223.174534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10233.530359] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.177830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10239.073656] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10241.043175] Lustre: Failing over lustre-MDT0000 [10241.467631] Lustre: server umount lustre-MDT0000 complete [10243.556680] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10243.580281] LustreError: 348341:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.586977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10243.603243] LustreError: 348341:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [10259.423618] Lustre: 340573:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787872350/real 1787872350] req@0000000023eb2ff1 x1874719695535488/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787872366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10259.458664] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10261.285964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10269.688046] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfbd56e5a3237e6ac [10269.705315] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [10269.721906] Lustre: Skipped 2 previous similar messages [10270.173961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10270.240821] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10271.721373] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10274.750663] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10275.301430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10275.339486] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10275.406144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10285.067203] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10286.786665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10295.788523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10295.789324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10295.807976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10295.808664] Lustre: Skipped 3 previous similar messages [10298.905739] Lustre: server umount lustre-MDT0000 complete [10300.896360] LustreError: 347082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.932105] LustreError: 347082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [10303.186404] LustreError: 347061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872410 with bad export cookie 18146531607227917996 [10303.223135] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10303.585891] Lustre: server umount lustre-MDT0001 complete [10318.162537] Lustre: server umount lustre-OST0000 complete [10334.596766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.156508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10339.865982] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10348.348928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.595701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10362.305965] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10369.415351] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10376.586406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10379.951971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10384.254828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10391.963586] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10403.815520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.816676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10403.828015] Lustre: Skipped 1 previous similar message [10403.832184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.843365] Lustre: Skipped 3 previous similar messages [10408.316169] Lustre: server umount lustre-OST0000 complete [10414.057507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.070590] Lustre: Skipped 1 previous similar message [10418.450251] Lustre: server umount lustre-MDT0000 complete [10419.170735] LustreError: 354185:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.189294] LustreError: 354185:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10422.698245] LustreError: 353437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872529 with bad export cookie 18146531607227918717 [10422.703080] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10422.723982] LustreError: 353437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10423.144315] Lustre: server umount lustre-MDT0001 complete [10437.076669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10440.223649] Key type lgssc unregistered [10440.680630] LNet: 356992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10440.685109] LNetError: 356992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10440.714182] LNet: Removed LNI 192.168.203.144@tcp [10441.827461] Key type .llcrypt unregistered [10441.831112] Key type ._llcrypt unregistered [10472.474246] Key type ._llcrypt registered [10472.476254] Key type .llcrypt registered [10472.587177] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10473.355323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10473.427478] alg: No test for adler32 (adler32-zlib) [10474.581437] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [10474.855650] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10476.543166] Key type lgssc registered [10478.162507] Lustre: Echo OBD driver; http://www.lustre.org/ [10487.867598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10487.888290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.323211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10494.187634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10499.555417] LustreError: 358511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.573533] LustreError: 358511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10504.338969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.673729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10504.841787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10510.404636] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10519.623220] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10526.091391] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10534.355061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10534.686777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10535.735241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10540.811939] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10548.568110] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10560.489066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10560.498926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10560.510612] Lustre: Skipped 1 previous similar message [10560.526289] Lustre: Skipped 2 previous similar messages [10561.561111] Lustre: server umount lustre-MDT0000 complete [10565.563288] LustreError: 358491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872672 with bad export cookie 12966926148493809988 [10565.599409] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.600228] LustreError: 358491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10565.621363] LustreError: 361006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10565.621941] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.676510] Lustre: Skipped 1 previous similar message [10565.688892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10566.123698] Lustre: server umount lustre-MDT0001 complete [10572.916262] Lustre: server umount lustre-OST0000 complete [10586.836960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.442461] LustreError: 362224:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.505262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10591.964482] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10592.736811] LustreError: 362225:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10597.862661] LustreError: 362224:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10601.551264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.065264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10607.154299] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10616.096370] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10623.145903] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10631.184596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10631.743438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10632.764263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10638.341816] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10646.545077] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10662.375201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.381778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10662.399049] Lustre: Skipped 2 previous similar messages [10662.404304] Lustre: Skipped 1 previous similar message [10664.937813] Lustre: server umount lustre-MDT0000 complete [10667.492876] LustreError: 362221:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10667.516258] LustreError: 362221:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10668.973334] LustreError: 362962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872775 with bad export cookie 12966926148493811108 [10668.974459] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10668.996233] LustreError: 362962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10669.413232] Lustre: server umount lustre-MDT0001 complete [10684.016164] Lustre: server umount lustre-OST0000 complete [10699.123550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.631403] LustreError: 366138:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10699.695205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10703.816343] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10712.187342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10717.856289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10727.306476] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10734.034695] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10740.870621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10741.137431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10741.144360] Lustre: Skipped 1 previous similar message [10744.243520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10747.011464] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10753.986806] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10759.522116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10759.526169] Lustre: Skipped 1 previous similar message [10759.531282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.718679] Lustre: server umount lustre-OST0000 complete [10764.776578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.789718] Lustre: Skipped 1 previous similar message [10768.800386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10768.815727] Lustre: Skipped 4 previous similar messages [10779.109393] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10779.445697] Lustre: server umount lustre-MDT0000 complete [10784.046523] LustreError: 366119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787872890 with bad export cookie 12966926148493812123 [10784.046968] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10784.056705] LustreError: 366119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10784.444290] Lustre: server umount lustre-MDT0001 complete [10795.389179] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10798.425808] Key type lgssc unregistered [10798.813715] LNet: 369672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10798.825444] LNetError: 369672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10798.844926] LNet: Removed LNI 192.168.203.144@tcp [10799.744086] Key type .llcrypt unregistered [10799.750550] Key type ._llcrypt unregistered [10824.546182] Key type ._llcrypt registered [10824.547796] Key type .llcrypt registered [10824.637093] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10825.458558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10825.482564] alg: No test for adler32 (adler32-zlib) [10826.579464] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [10826.868289] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10828.543296] Key type lgssc registered [10829.775914] Lustre: Echo OBD driver; http://www.lustre.org/ [10836.789818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10836.810585] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.817500] LustreError: 371115:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10836.836967] LustreError: 371115:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10836.842968] LustreError: 371115:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10836.848423] LustreError: 371115:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10836.860553] LustreError: 371115:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10843.793033] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10843.800639] LustreError: 371512:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10843.818381] LustreError: 371512:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10843.829359] LustreError: 371512:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10843.836292] LustreError: 371512:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10843.843387] LustreError: 371512:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10862.379302] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10865.239902] Key type lgssc unregistered [10865.559993] LNet: 372518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10865.567785] LNetError: 372518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10865.585129] LNet: Removed LNI 192.168.203.144@tcp [10866.265536] Key type .llcrypt unregistered [10866.268250] Key type ._llcrypt unregistered [10884.051374] Key type ._llcrypt registered [10884.054274] Key type .llcrypt registered [10884.196381] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10897.900255] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10898.987837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10899.006918] alg: No test for adler32 (adler32-zlib) [10900.068635] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [10900.315840] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10901.991584] Key type lgssc registered [10903.139641] Lustre: Echo OBD driver; http://www.lustre.org/ [10936.807341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10936.835203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.158794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10938.278737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10938.485738] Lustre: lustre-MDT0000: new disk, initializing [10938.793234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10938.827755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10944.644157] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10957.700742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.912929] Lustre: 375671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10958.059565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10958.064018] Lustre: Skipped 1 previous similar message [10958.178178] Lustre: lustre-MDT0001: new disk, initializing [10958.356525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10958.447769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10958.475099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10963.170984] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10973.595292] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10980.755695] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10987.794436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.159505] Lustre: lustre-OST0000: new disk, initializing [10988.170523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10988.179343] Lustre: 377644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10988.275959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10994.412210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10997.838888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10997.850883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10997.933232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11002.863208] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11007.211616] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11007.409359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11011.715842] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11011.952767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11018.722548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11018.724954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11018.732365] LustreError: Skipped 1 previous similar message [11018.735579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11018.747232] Lustre: Skipped 1 previous similar message [11019.939399] Lustre: server umount lustre-OST0000 complete [11023.846949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11023.853970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11023.858378] Lustre: Skipped 1 previous similar message [11023.868854] Lustre: Skipped 2 previous similar messages [11028.969393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11028.978785] Lustre: Skipped 1 previous similar message [11029.565353] Lustre: server umount lustre-MDT0000 complete [11033.321195] LustreError: 375663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873140 with bad export cookie 10443953982318555555 [11033.324920] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11033.334960] LustreError: 375663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11033.650946] Lustre: server umount lustre-MDT0001 complete [11042.079858] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11043.904486] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:25:48 (1787873148) [11050.487275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11051.080696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11055.440218] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11056.100285] LustreError: 380104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.115497] LustreError: 380104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11060.204098] LustreError: 380105:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11063.154172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.370177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11066.631333] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11074.126397] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11080.904305] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11089.120108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11089.526284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.578681] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11106.304590] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11113.901739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11114.174879] Lustre: lustre-OST0001: new disk, initializing [11114.183644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11114.192940] Lustre: 382940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11114.332340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11121.275281] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11123.838124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11123.848860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11123.930708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11130.939781] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11139.070871] Lustre: lustre-OST0001: Not available for connect from 192.168.203.44@tcp (stopping) [11139.564596] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11139.573721] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.600117] Lustre: server umount lustre-OST0001 complete [11144.163914] LustreError: 381926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.815462] LustreError: 382678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.838097] LustreError: 382678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11154.916975] LustreError: 381927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.937950] LustreError: 381927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11160.039927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.048324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11160.066051] Lustre: Skipped 2 previous similar messages [11160.075125] Lustre: Skipped 3 previous similar messages [11165.169818] LustreError: 381924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.203927] LustreError: 381924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11165.533848] Lustre: server umount lustre-OST0000 complete [11170.275303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11170.283520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11170.298392] Lustre: Skipped 1 previous similar message [11170.298642] Lustre: Skipped 2 previous similar messages [11175.885969] Lustre: server umount lustre-MDT0000 complete [11179.423990] LustreError: 380085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873286 with bad export cookie 10443953982318556437 [11179.434452] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.438204] LustreError: 380085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11185.951629] Lustre: server umount lustre-MDT0001 complete [11197.552450] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11200.559550] Key type lgssc unregistered [11200.856201] LNet: 384930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11200.863425] LNetError: 384930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11200.892528] LNet: Removed LNI 192.168.203.144@tcp [11201.805498] Key type .llcrypt unregistered [11201.813597] Key type ._llcrypt unregistered [11254.003822] Key type ._llcrypt registered [11254.009642] Key type .llcrypt registered [11254.214649] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11254.955335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11255.027918] alg: No test for adler32 (adler32-zlib) [11256.165200] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11256.388065] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11258.079934] Key type lgssc registered [11259.697830] Lustre: Echo OBD driver; http://www.lustre.org/ [11270.500241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11270.529394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11271.909274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11271.943363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11272.205736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11278.551342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11289.870568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11290.144848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11290.216210] Lustre: 387732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11290.596507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11295.273664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11303.007468] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11309.177390] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11315.685478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.693482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11315.706855] Lustre: Skipped 1 previous similar message [11317.747304] Lustre: server umount lustre-MDT0000 complete [11321.893564] LustreError: 387725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873428 with bad export cookie 10597296688023985095 [11321.929300] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11322.294694] Lustre: server umount lustre-MDT0001 complete [11332.141948] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:30:36 (1787873436) [11338.993354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11339.576129] LustreError: 390244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11339.591706] LustreError: 390244:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11339.644802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11343.899661] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11344.864516] LustreError: 390245:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11349.988347] LustreError: 390244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11352.891964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11353.312394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11358.527476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11366.742898] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11373.035507] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11380.365679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11380.621036] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11380.870492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11387.759593] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11395.998485] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11403.505789] Lustre: Permanently deactivating lustre-OST0000 [11403.522799] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11410.971819] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11412.974307] LustreError: 393229:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11412.983639] LustreError: 393229:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11412.989680] LustreError: 393229:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11415.640345] LustreError: 393278:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11415.647867] LustreError: 393278:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11423.805636] Lustre: server umount lustre-OST0000 complete [11427.006275] LustreError: 393527:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11427.015470] LustreError: 393527:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11430.368234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11430.376640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11430.391067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11430.399198] Lustre: Skipped 1 previous similar message [11436.004598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11436.010866] Lustre: Skipped 1 previous similar message [11440.156616] Lustre: server umount lustre-MDT0000 complete [11441.127525] LustreError: 390239:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.151139] LustreError: 390239:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11443.629080] LustreError: 390225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873550 with bad export cookie 10597296688023985676 [11443.632103] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11443.639144] LustreError: 390225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11454.105981] Lustre: server umount lustre-MDT0001 complete [11463.644546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11468.376795] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11471.468942] LustreError: 394686:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11478.454551] Lustre: 395026:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [11480.675981] Lustre: server umount MGS complete [11492.405255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11493.026206] LustreError: 395749:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11493.048279] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11493.055274] Lustre: Skipped 1 previous similar message [11493.147878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11497.690741] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11503.584417] LustreError: 395750:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11503.604879] LustreError: 395750:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11506.382333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11506.697220] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11511.925070] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11519.504319] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11525.997408] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11528.138730] Lustre: Permanently reactivating lustre-OST0000 [11528.169084] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11528.178574] Lustre: Skipped 2 previous similar messages [11555.295238] LustreError: 397522:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11563.199939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11563.554960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11563.567358] Lustre: Skipped 1 previous similar message [11569.841371] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11577.880184] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11584.999928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11585.019398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11585.048236] Lustre: Skipped 1 previous similar message [11585.058139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11585.786778] Lustre: server umount lustre-OST0000 complete [11591.136510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11591.139247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11591.150465] Lustre: Skipped 1 previous similar message [11599.459322] Lustre: server umount lustre-MDT0000 complete [11600.360704] LustreError: 395749:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11600.384735] LustreError: 395749:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11603.301787] LustreError: 395732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873710 with bad export cookie 10597296688023986726 [11603.311394] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11603.311770] LustreError: 395732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11603.622519] Lustre: server umount lustre-MDT0001 complete [11615.600849] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11618.606587] Key type lgssc unregistered [11618.851560] LNet: 400119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11618.857692] LNetError: 400119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11618.881438] LNet: Removed LNI 192.168.203.144@tcp [11619.724368] Key type .llcrypt unregistered [11619.731029] Key type ._llcrypt unregistered [11637.967455] Key type ._llcrypt registered [11637.975783] Key type .llcrypt registered [11638.128105] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11656.034390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11657.355688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11657.401881] alg: No test for adler32 (adler32-zlib) [11658.540635] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11658.841639] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11660.495245] Key type lgssc registered [11661.528642] Lustre: Echo OBD driver; http://www.lustre.org/ [11697.371933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11697.411284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.614874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11698.643792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11698.718416] Lustre: lustre-MDT0000: new disk, initializing [11698.848352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11698.874986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11703.173909] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11714.472107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.597390] Lustre: 403276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11714.681911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11714.687503] Lustre: Skipped 1 previous similar message [11714.765812] Lustre: lustre-MDT0001: new disk, initializing [11714.865090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11714.903425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11714.924519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11719.423460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11730.437225] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11737.226287] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11744.845292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11745.139872] Lustre: lustre-OST0000: new disk, initializing [11745.149129] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11745.162277] Lustre: 405246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11745.262069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11752.446734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11755.085820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11755.097713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11755.217050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11762.678844] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11767.705617] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11767.946740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11772.261479] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11772.563136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11775.971982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11775.982358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11775.998568] Lustre: Skipped 1 previous similar message [11776.012049] Lustre: Skipped 1 previous similar message [11780.579571] Lustre: server umount lustre-OST0000 complete [11786.213042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.219829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11786.229687] Lustre: Skipped 1 previous similar message [11786.246471] Lustre: Skipped 1 previous similar message [11790.493538] Lustre: server umount lustre-MDT0000 complete [11794.296858] LustreError: 405245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873901 with bad export cookie 10967009526900651790 [11794.303148] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11794.318083] LustreError: 405245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11794.674774] Lustre: server umount lustre-MDT0001 complete [11803.945220] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11805.744543] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:38:30 (1787873910) [11812.639313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.923127] LustreError: 407707:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11812.932907] LustreError: 407707:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11812.985439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11817.136422] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11818.468290] LustreError: 407708:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.561244] LustreError: 407707:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11825.683110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11825.891297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11830.341390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11838.123871] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11843.447804] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11851.225772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.763437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11858.263942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11866.056276] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11868.315751] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11870.947423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11878.374919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11878.405532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.447243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11880.416872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11883.679489] Lustre: server umount lustre-OST0000 complete [11890.669147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.681641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11890.685284] Lustre: Skipped 1 previous similar message [11890.713314] Lustre: Skipped 3 previous similar messages [11894.043601] Lustre: server umount lustre-MDT0000 complete [11895.779143] LustreError: 408436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.808140] LustreError: 408436:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11898.047729] LustreError: 407689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874004 with bad export cookie 10967009526900652672 [11898.050733] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.059162] LustreError: 407689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11898.449361] Lustre: server umount lustre-MDT0001 complete [11909.780752] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11912.507859] Key type lgssc unregistered [11912.835718] LNet: 411438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11912.851772] LNetError: 411438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11913.899842] LNet: Removed LNI 192.168.203.144@tcp [11914.632626] Key type .llcrypt unregistered [11914.636698] Key type ._llcrypt unregistered [11939.058401] Key type ._llcrypt registered [11939.061874] Key type .llcrypt registered [11939.222820] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11939.608683] Key type .llcrypt unregistered [11939.610637] Key type ._llcrypt unregistered [11957.308377] Key type ._llcrypt registered [11957.310381] Key type .llcrypt registered [11957.418866] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11958.712224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11958.737290] alg: No test for adler32 (adler32-zlib) [11959.826960] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11960.153318] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11961.888061] Key type lgssc registered [11963.156522] Lustre: Echo OBD driver; http://www.lustre.org/ [11971.261286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11971.278742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.835728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11976.922217] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11982.312087] LustreError: 413670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.342029] LustreError: 413670:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11986.007864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11986.290284] LustreError: 413671:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.320458] LustreError: 413671:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11986.329359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11991.525361] LustreError: 413670:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.556655] LustreError: 413670:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11992.195361] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11996.653878] LustreError: 413667:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.689567] LustreError: 413667:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11997.749455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11999.625717] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12002.791499] LustreError: 413666:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12002.819647] LustreError: 413666:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12005.777147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12006.148106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12010.714367] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12021.335402] Lustre: server umount lustre-OST0000 complete [12026.849724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.850967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12026.862755] Lustre: Skipped 1 previous similar message [12026.872257] Lustre: Skipped 1 previous similar message [12031.970250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12031.982542] Lustre: Skipped 1 previous similar message [12035.103251] Lustre: server umount lustre-MDT0000 complete [12037.093275] LustreError: 413665:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.120227] LustreError: 413665:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12038.112716] Lustre: 412979:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787874128/real 1787874128] req@00000000229824fb x1874721836656256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787874144 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12038.147898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12039.248845] LustreError: 413650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874146 with bad export cookie 11995255303760045481 [12039.250581] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12039.266720] LustreError: 413650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12039.617490] Lustre: server umount lustre-MDT0001 complete [12049.599841] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12052.207115] Key type lgssc unregistered [12052.527621] LNet: 416818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12052.531518] LNetError: 416818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12052.541189] LNet: Removed LNI 192.168.203.144@tcp [12053.268627] Key type .llcrypt unregistered [12053.270716] Key type ._llcrypt unregistered [12075.691840] Key type ._llcrypt registered [12075.696608] Key type .llcrypt registered [12075.808761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12076.627893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12076.670889] alg: No test for adler32 (adler32-zlib) [12077.731848] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12077.913696] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12079.599310] Key type lgssc registered [12080.629206] Lustre: Echo OBD driver; http://www.lustre.org/ [12087.881252] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12087.913494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12103.519566] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12108.639839] LustreError: 418208:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [12135.327178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12142.095793] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12150.171300] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12157.318996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12157.971067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12158.075078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12162.478245] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12163.056660] LustreError: 418234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12163.095588] LustreError: 418234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12168.172204] LustreError: 418233:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.193459] LustreError: 418234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.201876] LustreError: 418234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12172.087330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.339469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12178.257509] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12186.505819] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12192.937990] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12202.988172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.991821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12202.995666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12203.013256] Lustre: Skipped 1 previous similar message [12206.823045] Lustre: server umount lustre-OST0000 complete [12213.223132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.223575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12213.236619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12213.236627] Lustre: Skipped 1 previous similar message [12213.240403] Lustre: Skipped 1 previous similar message [12216.898798] Lustre: server umount lustre-MDT0000 complete [12218.339872] LustreError: 419252:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.356826] LustreError: 419252:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12220.900601] LustreError: 418215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874327 with bad export cookie 5091922345702307604 [12220.913748] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12220.923911] LustreError: 418215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12221.494312] Lustre: server umount lustre-MDT0001 complete [12232.081660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12234.837694] Key type lgssc unregistered [12235.148273] LNet: 421783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12235.156879] LNetError: 421783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12235.177910] LNet: Removed LNI 192.168.203.144@tcp [12236.090929] Key type .llcrypt unregistered [12236.096434] Key type ._llcrypt unregistered [12258.711761] Key type ._llcrypt registered [12258.716906] Key type .llcrypt registered [12258.810979] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12259.810632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12259.943943] alg: No test for adler32 (adler32-zlib) [12260.988146] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12261.247963] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12262.935880] Key type lgssc registered [12264.085346] Lustre: Echo OBD driver; http://www.lustre.org/ [12270.457731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12270.507608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12271.870876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12277.054833] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12281.314251] LustreError: 423193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12281.350514] LustreError: 423193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12285.087977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12285.324150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12289.612264] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12297.521475] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12303.605511] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12309.722757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12310.189961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12316.351419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12316.529657] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12324.182168] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12331.489326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12331.500228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.519734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12332.003057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.021044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12332.029115] Lustre: Skipped 1 previous similar message [12335.346929] Lustre: server umount lustre-MDT0000 complete [12337.122041] LustreError: 423193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.151222] LustreError: 423193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12337.645976] LustreError: 423175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.44@tcp arrived at 1787874444 with bad export cookie 11284542372293693766 [12337.677407] LustreError: 423175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12339.462764] LustreError: 423173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787874446 with bad export cookie 11284542372293692870 [12339.469078] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12339.477417] LustreError: 423173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12342.240703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12342.242909] LustreError: 424434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.248730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12342.248739] Lustre: Skipped 1 previous similar message [12342.254053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12342.276317] LustreError: 424434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12342.302258] Lustre: Skipped 1 previous similar message [12344.288992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12344.290977] LustreError: 423189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.313839] LustreError: 423189:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12345.760496] Lustre: server umount lustre-OST0000 complete [12349.408926] LustreError: 423190:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12349.434743] LustreError: 423190:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12352.281466] Lustre: server umount lustre-MDT0001 complete [12363.521929] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12366.116772] Key type lgssc unregistered [12366.492795] LNet: 426882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12366.508415] LNetError: 426882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12366.529931] LNet: Removed LNI 192.168.203.144@tcp [12367.402702] Key type .llcrypt unregistered [12367.405769] Key type ._llcrypt unregistered [12392.490138] Key type ._llcrypt registered [12392.492380] Key type .llcrypt registered [12392.630523] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12393.759487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12393.774474] alg: No test for adler32 (adler32-zlib) [12394.964325] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12395.365124] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12397.034481] Key type lgssc registered [12398.301491] Lustre: Echo OBD driver; http://www.lustre.org/ [12405.157840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12405.173782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12406.561952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12410.970248] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12417.006033] LustreError: 428302:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12417.051162] LustreError: 428302:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12420.164328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.588727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12425.490608] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12433.388305] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12439.866468] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12446.705055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12447.049780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12451.198935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12454.020385] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12462.464950] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12470.753452] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12470.764593] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [12470.816641] Lustre: Skipped 2 previous similar messages [12473.777508] Lustre: server umount lustre-MDT0001 complete [12475.874619] LustreError: 430141:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.905195] LustreError: 430141:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12480.994306] LustreError: 429030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12481.016311] LustreError: 429030:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12482.017408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12482.039749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12482.070105] Lustre: Skipped 1 previous similar message [12482.078944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12486.121983] LustreError: 428302:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12486.124943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12487.984594] Lustre: server umount lustre-OST0000 complete [12491.247307] LustreError: 429030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12492.609843] Lustre: server umount lustre-MDT0000 complete [12505.239271] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12507.515330] Key type lgssc unregistered [12507.743591] LNet: 431992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12507.778169] LNetError: 431992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12507.798767] LNet: Removed LNI 192.168.203.144@tcp [12508.517140] Key type .llcrypt unregistered [12508.520227] Key type ._llcrypt unregistered [12538.213136] Key type ._llcrypt registered [12538.221552] Key type .llcrypt registered [12538.385443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12553.658949] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12555.219934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12555.279300] alg: No test for adler32 (adler32-zlib) [12556.378831] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12556.699610] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12558.415157] Key type lgssc registered [12559.610594] Lustre: Echo OBD driver; http://www.lustre.org/ [12606.681859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12621.735553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12621.761522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12623.043819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12623.082833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12623.166512] Lustre: lustre-MDT0000: new disk, initializing [12623.240714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12623.255821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12627.334264] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12641.692057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12641.984671] Lustre: lustre-OST0000: new disk, initializing [12641.988941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12641.993878] Lustre: Skipped 1 previous similar message [12642.000703] Lustre: 437028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12642.101185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12648.076942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12650.075519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12650.083272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12650.121329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12657.578723] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12672.953526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12673.131783] Lustre: 436012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12673.247654] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12673.367958] Lustre: lustre-MDT0001: new disk, initializing [12673.506878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12673.535440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12673.558448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12678.091947] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12687.306585] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12687.565144] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12691.817138] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12692.000683] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12700.150355] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [12700.645211] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12700.647636] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12700.653166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12700.672104] Lustre: Skipped 1 previous similar message [12702.848313] Lustre: server umount lustre-MDT0001 complete [12710.880438] LustreError: 436023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.894484] LustreError: 436023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12716.007021] LustreError: 436022:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.034395] LustreError: 436022:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12719.016610] LustreError: 436023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.035225] LustreError: 436023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12721.637590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12721.642650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12721.650561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12721.654505] Lustre: Skipped 1 previous similar message [12724.196081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12724.198644] LustreError: 436022:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12726.092534] Lustre: server umount lustre-OST0000 complete [12729.316145] LustreError: 436023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12736.523150] Lustre: server umount lustre-MDT0000 complete [12751.228982] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12754.082220] Key type lgssc unregistered [12754.361973] LNet: 440780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12754.370278] LNetError: 440780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12754.387756] LNet: Removed LNI 192.168.203.144@tcp [12755.302850] Key type .llcrypt unregistered [12755.305739] Key type ._llcrypt unregistered [12795.894552] Key type ._llcrypt registered [12795.898812] Key type .llcrypt registered [12795.982916] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12796.261177] Key type .llcrypt unregistered [12796.267169] Key type ._llcrypt unregistered [12812.732641] Key type ._llcrypt registered [12812.738220] Key type .llcrypt registered [12812.814589] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12813.747798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12813.779200] alg: No test for adler32 (adler32-zlib) [12814.798744] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12814.992170] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12816.663176] Key type lgssc registered [12817.515203] Lustre: Echo OBD driver; http://www.lustre.org/ [12848.796887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12848.882681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12850.354235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12850.407364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12850.525654] Lustre: lustre-MDT0000: new disk, initializing [12850.680053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12850.711345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12854.840266] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12865.607697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12865.720393] Lustre: 444166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12865.773459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12865.784388] Lustre: Skipped 1 previous similar message [12865.896594] Lustre: lustre-MDT0001: new disk, initializing [12866.030682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12866.066897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12866.072887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12871.608764] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12881.473587] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12888.344992] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12895.393922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12895.725946] Lustre: lustre-OST0000: new disk, initializing [12895.728648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12895.739983] Lustre: 446135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12895.862894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12901.117119] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12903.504851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12903.511976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12903.673609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12909.027661] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12917.219107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12917.222971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.235477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12919.266575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.267956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12919.288579] Lustre: Skipped 2 previous similar messages [12920.702072] Lustre: server umount lustre-MDT0000 complete [12923.874343] LustreError: 444155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875030 with bad export cookie 9559867456353209937 [12923.878972] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12923.892820] LustreError: 444155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12924.154353] Lustre: server umount lustre-MDT0001 complete [12937.832178] Lustre: server umount lustre-OST0000 complete [12947.564880] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:57:32 (1787875052) [12954.591988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12955.138488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12959.901409] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12960.251553] LustreError: 448159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12960.275107] LustreError: 448159:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12964.388591] LustreError: 448160:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12969.031985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12969.560591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12974.438979] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12982.624257] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12987.766940] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13008.299063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13008.646755] Lustre: Found index 0 for lustre-OST0000, updating log [13008.915266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13015.167976] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13021.404495] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13026.349181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13035.488606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13035.499670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.510189] Lustre: Skipped 1 previous similar message [13035.514823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13042.656983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13042.673429] Lustre: Skipped 4 previous similar messages [13046.239151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13046.496347] Lustre: server umount lustre-OST0000 complete [13064.162551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.166493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13064.186859] Lustre: Skipped 2 previous similar messages [13064.200087] Lustre: Skipped 1 previous similar message [13067.916120] Lustre: server umount lustre-MDT0000 complete [13069.281955] LustreError: 449152:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13069.312288] LustreError: 449152:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13071.087454] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875177 with bad export cookie 9559867456353211330 [13071.094887] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13071.098052] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13071.326823] Lustre: server umount lustre-MDT0001 complete [13078.175393] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13082.628173] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:59:47 (1787875187) [13703.352519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13703.723642] LustreError: 454582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.830403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13708.304546] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13709.281255] LustreError: 454583:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13713.388523] LustreError: 454582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13716.210823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13716.557617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13720.855602] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13727.787757] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13733.678934] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13752.818136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13753.013978] Lustre: Found index 0 for lustre-OST0000, updating log [13753.208746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13755.167487] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13757.844232] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13764.648931] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13766.600843] Lustre: server umount lustre-OST0000 complete [13771.561163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13771.565153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13771.590616] Lustre: Skipped 1 previous similar message [13771.616653] Lustre: Skipped 1 previous similar message [13772.647877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13772.660953] Lustre: Skipped 1 previous similar message [13774.690855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13774.699414] Lustre: Skipped 3 previous similar messages [13776.800551] Lustre: server umount lustre-MDT0000 complete [13777.773782] LustreError: 455128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13777.795949] LustreError: 455128:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13781.261906] LustreError: 454563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875888 with bad export cookie 9559867456353212408 [13781.270506] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13781.273863] LustreError: 454563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13781.522787] Lustre: server umount lustre-MDT0001 complete [13810.651346] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13815.583666] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [13825.970980] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13873.011840] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:12:57 (1787875977) [13897.375310] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13901.246867] Key type lgssc unregistered [13901.679366] LNet: 461212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13901.693529] LNetError: 461212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14014.374624] LNet: Removed LNI 192.168.203.144@tcp [14015.036344] Key type .llcrypt unregistered [14015.040661] Key type ._llcrypt unregistered [14032.167169] Key type ._llcrypt registered [14032.170399] Key type .llcrypt registered [14032.317197] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14033.689533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14033.730514] alg: No test for adler32 (adler32-zlib) [14035.012379] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [14035.408480] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14037.184353] Key type lgssc registered [14039.410458] Lustre: Echo OBD driver; http://www.lustre.org/ [14045.887168] LNet: Added LNI 192.168.203.144@tcp100 [8/256/0/180] [14046.742370] LNet: Added LNI 192.168.203.144@tcp101 [8/256/0/180] [14047.838815] LNet: Added LNI 192.168.203.144@tcp102 [8/256/0/180] [14048.776565] LNet: Added LNI 192.168.203.144@tcp103 [8/256/0/180] [14050.098952] LNet: Added LNI 192.168.203.144@tcp104 [8/256/0/180] [14051.341692] LNet: Added LNI 192.168.203.144@tcp105 [8/256/0/180] [14052.373737] LNet: Added LNI 192.168.203.144@tcp106 [8/256/0/180] [14053.568107] LNet: Added LNI 192.168.203.144@tcp107 [8/256/0/180] [14054.959915] LNet: Added LNI 192.168.203.144@tcp108 [8/256/0/180] [14055.891045] LNet: Added LNI 192.168.203.144@tcp109 [8/256/0/180] [14057.124053] LNet: Added LNI 192.168.203.144@tcp110 [8/256/0/180] [14058.174693] LNet: Added LNI 192.168.203.144@tcp111 [8/256/0/180] [14059.343785] LNet: Added LNI 192.168.203.144@tcp112 [8/256/0/180] [14060.617272] LNet: Added LNI 192.168.203.144@tcp113 [8/256/0/180] [14061.785738] LNet: Added LNI 192.168.203.144@tcp114 [8/256/0/180] [14062.846942] LNet: Added LNI 192.168.203.144@tcp115 [8/256/0/180] [14063.780190] LNet: Added LNI 192.168.203.144@tcp116 [8/256/0/180] [14064.637441] LNet: Added LNI 192.168.203.144@tcp117 [8/256/0/180] [14065.401629] LNet: Added LNI 192.168.203.144@tcp118 [8/256/0/180] [14066.640861] LNet: Added LNI 192.168.203.144@tcp119 [8/256/0/180] [14067.645648] LNet: Added LNI 192.168.203.144@tcp120 [8/256/0/180] [14068.550717] LNet: Added LNI 192.168.203.144@tcp121 [8/256/0/180] [14069.554444] LNet: Added LNI 192.168.203.144@tcp122 [8/256/0/180] [14070.562945] LNet: Added LNI 192.168.203.144@tcp123 [8/256/0/180] [14071.505277] LNet: Added LNI 192.168.203.144@tcp124 [8/256/0/180] [14072.381390] LNet: Added LNI 192.168.203.144@tcp125 [8/256/0/180] [14073.171586] LNet: Added LNI 192.168.203.144@tcp126 [8/256/0/180] [14073.946914] LNet: Added LNI 192.168.203.144@tcp127 [8/256/0/180] [14074.801296] LNet: Added LNI 192.168.203.144@tcp128 [8/256/0/180] [14075.499559] LNet: Added LNI 192.168.203.144@tcp129 [8/256/0/180] [14076.404823] LNet: Added LNI 192.168.203.144@tcp130 [8/256/0/180] [14077.289320] LNet: Added LNI 192.168.203.144@tcp131 [8/256/0/180] [14078.105182] LNet: Added LNI 192.168.203.144@tcp132 [8/256/0/180] [14079.097430] LNet: Added LNI 192.168.203.144@tcp133 [8/256/0/180] [14079.990763] LNet: Added LNI 192.168.203.144@tcp134 [8/256/0/180] [14080.893974] LNet: Added LNI 192.168.203.144@tcp135 [8/256/0/180] [14088.663778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14088.686084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14089.769875] Lustre: 464243:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14089.875631] LNetError: 461699:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14089.889900] LNetError: Protocol error connecting to 192.168.203.144@tcp121 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14089.903508] LNetError: Skipped 1 previous similar message [14089.985601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14090.018479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14090.146973] Lustre: lustre-MDT0000: new disk, initializing [14090.311727] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14090.331223] Lustre: 464300:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14090.364326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14090.393443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14090.724645] LNet: 6 local NIs in recovery (showing 6): 192.168.203.144@tcp122, 192.168.203.144@tcp124, 192.168.203.144@tcp125, 192.168.203.144@tcp121, 192.168.203.144@tcp127, 192.168.203.144@tcp123 [14090.729175] LNetError: 461700:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14090.767939] LNetError: 461700:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14090.774420] LNetError: Protocol error connecting to 192.168.203.144@tcp108 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14090.783872] LNetError: Skipped 5 previous similar messages [14095.790484] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14107.427926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14107.524415] Lustre: 464262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14107.592761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14107.597234] Lustre: Skipped 1 previous similar message [14107.689824] Lustre: lustre-MDT0001: new disk, initializing [14107.745469] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14107.752150] Lustre: 465062:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14107.938265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14108.001291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14108.017520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14112.890907] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14115.434202] Lustre: 465421:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14115.454173] Lustre: 465421:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14119.397113] LNetError: 464268:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14119.411135] LNetError: Protocol error connecting to 192.168.203.144@tcp119 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14122.467357] LNetError: 461702:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14122.491950] LNetError: Protocol error connecting to 192.168.203.144@tcp117 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14124.471449] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14129.632420] LNetError: 461701:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14129.645938] LNetError: 461701:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14129.654263] LNetError: Protocol error connecting to 192.168.203.144@tcp108 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14129.669375] LNetError: Skipped 1 previous similar message [14131.842436] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14139.549285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14139.832787] Lustre: lustre-OST0000: new disk, initializing [14139.840550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14139.850618] Lustre: 466237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14139.930511] Lustre: 466243:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.144@tcp: can't add NID 192.168.203.144@tcp130: rc = -75 [14139.937495] Lustre: 466243:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14140.008416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14142.959975] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14142.966721] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14142.983946] LNetError: Protocol error connecting to 192.168.203.144@tcp108 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14142.998484] LNetError: Skipped 3 previous similar messages [14145.655934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14145.676428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14145.811039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14147.007414] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14156.764785] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14160.355944] LNetError: 461701:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.144 [14160.374065] LNetError: 461701:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14160.380613] LNetError: Protocol error connecting to 192.168.203.144@tcp117 at host 192.168.203.144:988: is it running a compatible version of Lustre? [14160.395837] LNetError: Skipped 5 previous similar messages [14166.503048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14166.503947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14166.518615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14166.550152] Lustre: Skipped 1 previous similar message [14166.573404] Lustre: Skipped 1 previous similar message [14171.619183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14171.631637] Lustre: Skipped 1 previous similar message [14176.755185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14176.761706] Lustre: Skipped 1 previous similar message [14180.321812] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14180.583461] Lustre: server umount lustre-OST0000 complete [14186.983445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14186.984336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14186.992493] Lustre: Skipped 1 previous similar message [14187.004622] Lustre: Skipped 1 previous similar message [14191.306472] Lustre: server umount lustre-MDT0000 complete [14195.121706] LustreError: 464252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876301 with bad export cookie 3018578363898909668 [14195.131762] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14195.138497] LustreError: 464252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14195.501963] Lustre: server umount lustre-MDT0001 complete [14206.515738] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14208.865374] Key type lgssc unregistered [14209.253459] LNet: 468154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14209.261945] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14209.269290] LNet: Removed LNI 192.168.203.144@tcp [14210.335569] LNet: Removed LNI 192.168.203.144@tcp100 [14210.350330] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14210.382041] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14210.401940] LNet: Removed LNI 192.168.203.144@tcp101 [14211.423152] LNet: Removed LNI 192.168.203.144@tcp102 [14211.431862] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14211.438046] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14211.446605] LNet: Removed LNI 192.168.203.144@tcp103 [14212.511173] LNet: Removed LNI 192.168.203.144@tcp104 [14213.535196] LNet: Removed LNI 192.168.203.144@tcp105 [14213.538224] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14213.545178] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14213.562582] LNet: Removed LNI 192.168.203.144@tcp106 [14214.623140] LNet: Removed LNI 192.168.203.144@tcp107 [14215.647283] LNet: Removed LNI 192.168.203.144@tcp108 [14216.671200] LNet: Removed LNI 192.168.203.144@tcp109 [14217.695317] LNet: Removed LNI 192.168.203.144@tcp110 [14217.701520] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14217.721566] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14217.739871] LNet: Removed LNI 192.168.203.144@tcp111 [14218.783203] LNet: Removed LNI 192.168.203.144@tcp112 [14219.807153] LNet: Removed LNI 192.168.203.144@tcp113 [14220.832109] LNet: Removed LNI 192.168.203.144@tcp114 [14221.855664] LNet: Removed LNI 192.168.203.144@tcp115 [14222.879879] LNet: Removed LNI 192.168.203.144@tcp116 [14223.903543] LNet: Removed LNI 192.168.203.144@tcp117 [14224.930892] LNet: Removed LNI 192.168.203.144@tcp118 [14225.951255] LNet: Removed LNI 192.168.203.144@tcp119 [14225.956828] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14225.966972] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14225.981366] LNet: Removed LNI 192.168.203.144@tcp120 [14227.039375] LNet: Removed LNI 192.168.203.144@tcp121 [14228.063424] LNet: Removed LNI 192.168.203.144@tcp122 [14229.087158] LNet: Removed LNI 192.168.203.144@tcp123 [14230.111191] LNet: Removed LNI 192.168.203.144@tcp124 [14231.135146] LNet: Removed LNI 192.168.203.144@tcp125 [14231.148693] LNet: Removed LNI 192.168.203.144@tcp126 [14232.224092] LNet: Removed LNI 192.168.203.144@tcp127 [14233.250211] LNet: Removed LNI 192.168.203.144@tcp128 [14234.335220] LNet: Removed LNI 192.168.203.144@tcp129 [14234.338329] LNet: Removed LNI 192.168.203.144@tcp130 [14234.341596] LNet: Removed LNI 192.168.203.144@tcp131 [14234.344483] LNet: Removed LNI 192.168.203.144@tcp132 [14234.350450] LNet: Removed LNI 192.168.203.144@tcp133 [14234.363116] LNet: Removed LNI 192.168.203.144@tcp134 [14234.373920] LNet: Removed LNI 192.168.203.144@tcp135 [14235.061530] Key type .llcrypt unregistered [14235.063416] Key type ._llcrypt unregistered [14256.722853] Key type ._llcrypt registered [14256.725128] Key type .llcrypt registered [14256.804491] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [14276.429575] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14278.345184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14278.489822] alg: No test for adler32 (adler32-zlib) [14279.552766] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [14279.871527] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14281.615173] Key type lgssc registered [14283.061939] Lustre: Echo OBD driver; http://www.lustre.org/ [14320.356640] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:20:24 (1787876424) [14341.755531] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14344.956457] Key type lgssc unregistered [14345.419886] LNet: 472248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14345.430983] LNetError: 472248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14345.449776] LNet: Removed LNI 192.168.203.144@tcp [14346.318319] Key type .llcrypt unregistered [14346.320995] Key type ._llcrypt unregistered [14367.544155] Key type ._llcrypt registered [14367.548243] Key type .llcrypt registered [14367.665986] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14369.097826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14369.132369] alg: No test for adler32 (adler32-zlib) [14370.317034] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [14370.567398] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14372.199193] Key type lgssc registered [14373.252912] Lustre: Echo OBD driver; http://www.lustre.org/ [14383.228137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14383.266633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14384.564258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14384.598288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14384.760612] Lustre: lustre-MDT0000: new disk, initializing [14384.851022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14384.866570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14390.014387] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14399.282847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14399.397055] Lustre: 473464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14399.436990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14399.443816] Lustre: Skipped 1 previous similar message [14399.501511] Lustre: lustre-MDT0001: new disk, initializing [14399.585690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14399.631627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14399.643238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14403.074703] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14413.276333] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14420.676537] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14430.229271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14430.566553] Lustre: lustre-OST0000: new disk, initializing [14430.571055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14430.581062] Lustre: 475436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14430.723625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14437.746762] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14440.021957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14440.029511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14440.157914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14448.184595] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14452.346518] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing lnet_if_list [14458.624482] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14458.668282] LNet: Added LNI 192.168.203.144@tcp10 [8/256/0/180] [14465.434569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14465.466903] LNet: Added LNI 192.168.203.144@tcp11 [8/256/0/180] [14470.534131] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14470.565916] LNet: Added LNI 192.168.203.144@tcp12 [8/256/0/180] [14475.246792] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14475.289175] LNet: Added LNI 192.168.203.144@tcp13 [8/256/0/180] [14480.513108] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14480.541598] LNet: Added LNI 192.168.203.144@tcp14 [8/256/0/180] [14485.374478] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14485.431358] LNet: Added LNI 192.168.203.144@tcp15 [8/256/0/180] [14490.462992] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14490.491910] LNet: Added LNI 192.168.203.144@tcp16 [8/256/0/180] [14496.060179] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14496.092441] LNet: Added LNI 192.168.203.144@tcp17 [8/256/0/180] [14501.761316] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14501.796188] LNet: Added LNI 192.168.203.144@tcp18 [8/256/0/180] [14507.257489] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14507.281718] LNet: Added LNI 192.168.203.144@tcp19 [8/256/0/180] [14512.848854] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14512.864592] LNet: Added LNI 192.168.203.144@tcp20 [8/256/0/180] [14517.859177] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14517.892936] LNet: Added LNI 192.168.203.144@tcp21 [8/256/0/180] [14524.389695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14524.422808] LNet: Added LNI 192.168.203.144@tcp22 [8/256/0/180] [14531.964896] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14532.021709] LNet: Added LNI 192.168.203.144@tcp23 [8/256/0/180] [14537.492779] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14537.518562] LNet: Added LNI 192.168.203.144@tcp24 [8/256/0/180] [14545.043914] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14545.105365] LNet: Added LNI 192.168.203.144@tcp25 [8/256/0/180] [14551.068749] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14551.106527] LNet: Added LNI 192.168.203.144@tcp26 [8/256/0/180] [14557.734583] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14557.786400] LNet: Added LNI 192.168.203.144@tcp27 [8/256/0/180] [14564.189540] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14564.249585] LNet: Added LNI 192.168.203.144@tcp28 [8/256/0/180] [14571.585637] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14571.613981] LNet: Added LNI 192.168.203.144@tcp29 [8/256/0/180] [14578.690804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14578.746106] LNet: Added LNI 192.168.203.144@tcp30 [8/256/0/180] [14585.393403] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14585.426938] LNet: Added LNI 192.168.203.144@tcp31 [8/256/0/180] [14591.576971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14591.618894] LNet: Added LNI 192.168.203.144@tcp32 [8/256/0/180] [14598.488539] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14598.519734] LNet: Added LNI 192.168.203.144@tcp33 [8/256/0/180] [14605.973146] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14606.014865] LNet: Added LNI 192.168.203.144@tcp34 [8/256/0/180] [14615.290382] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14615.327771] LNet: Added LNI 192.168.203.144@tcp35 [8/256/0/180] [14622.293244] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14622.313134] LNet: Added LNI 192.168.203.144@tcp36 [8/256/0/180] [14629.417717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14629.452831] LNet: Added LNI 192.168.203.144@tcp37 [8/256/0/180] [14638.437337] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14638.473643] LNet: Added LNI 192.168.203.144@tcp38 [8/256/0/180] [14648.925083] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14648.991684] LNet: Added LNI 192.168.203.144@tcp39 [8/256/0/180] [14659.956758] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14660.000431] LNet: Added LNI 192.168.203.144@tcp40 [8/256/0/180] [14667.574403] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14667.616160] LNet: Added LNI 192.168.203.144@tcp41 [8/256/0/180] [14675.259234] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14675.289504] LNet: Added LNI 192.168.203.144@tcp42 [8/256/0/180] [14683.072439] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14683.117225] LNet: Added LNI 192.168.203.144@tcp43 [8/256/0/180] [14690.449399] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14690.468439] LNet: Added LNI 192.168.203.144@tcp44 [8/256/0/180] [14700.014149] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14700.073889] LNet: Added LNI 192.168.203.144@tcp45 [8/256/0/180] [14710.643756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14710.720048] LNet: Added LNI 192.168.203.144@tcp46 [8/256/0/180] [14719.821567] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14719.874565] LNet: Added LNI 192.168.203.144@tcp47 [8/256/0/180] [14727.476423] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14727.529974] LNet: Added LNI 192.168.203.144@tcp48 [8/256/0/180] [14736.949425] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14737.018874] LNet: Added LNI 192.168.203.144@tcp49 [8/256/0/180] [14746.243657] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14746.270424] LNet: Added LNI 192.168.203.144@tcp50 [8/256/0/180] [14768.097905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14768.108397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14768.110452] Lustre: Skipped 1 previous similar message [14768.120700] Lustre: Skipped 2 previous similar messages [14771.801344] Lustre: server umount lustre-MDT0000 complete [14776.914187] LustreError: 475435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876883 with bad export cookie 7116971091574713354 [14776.931369] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14777.393346] Lustre: server umount lustre-MDT0001 complete [14794.655888] Lustre: 472736:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876885/real 1787876885] req@00000000b3693513 x1874724364923520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876901 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14794.687407] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14794.700564] Lustre: Skipped 1 previous similar message [14797.801343] Lustre: 472739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876888/real 1787876888] req@0000000026012c01 x1874724364923648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14799.854201] Lustre: 472738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876890/real 1787876890] req@00000000955fff65 x1874724364923776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14799.970654] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14802.848196] Lustre: 472736:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876893/real 1787876893] req@00000000ce30fe14 x1874724364924032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14808.033237] Lustre: 472737:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787876898/real 1787876898] req@00000000b03edffc x1874724364924160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787876914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14808.959856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14821.346957] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x62c48f801e08f644 [14821.360763] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [14821.583483] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14821.599211] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14821.620820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14823.652183] LustreError: 475430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14823.687474] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14824.681621] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14827.643342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14830.052071] LustreError: 475430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14835.175600] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14838.090338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14838.381054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14839.404476] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14843.449623] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [14852.037975] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14857.750636] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14880.740218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14880.745847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14880.751056] Lustre: Skipped 1 previous similar message [14880.765060] Lustre: Skipped 1 previous similar message [14885.720586] Lustre: server umount lustre-OST0000 complete [14890.976797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14890.979454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14890.985703] Lustre: Skipped 1 previous similar message [14896.097839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14896.104429] Lustre: Skipped 2 previous similar messages [14896.411333] Lustre: server umount lustre-MDT0000 complete [14900.198541] LustreError: 499683:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14900.204531] LustreError: 499683:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14901.552114] LustreError: 475435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877008 with bad export cookie 7116971091574715972 [14901.560981] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14901.570575] LustreError: 475435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14902.084492] Lustre: server umount lustre-MDT0001 complete [14915.570473] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [14918.116633] Key type lgssc unregistered [14918.678488] LNet: 502488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14918.697112] LNetError: 502488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14918.714091] LNet: Removed LNI 192.168.203.144@tcp [14918.720504] LNet: Removed LNI 192.168.203.144@tcp10 [14918.726502] LNet: Removed LNI 192.168.203.144@tcp11 [14918.731737] LNet: Removed LNI 192.168.203.144@tcp12 [14918.741267] LNet: Removed LNI 192.168.203.144@tcp13 [14918.748551] LNet: Removed LNI 192.168.203.144@tcp14 [14918.756565] LNet: Removed LNI 192.168.203.144@tcp15 [14918.765773] LNet: Removed LNI 192.168.203.144@tcp16 [14918.770907] LNet: Removed LNI 192.168.203.144@tcp17 [14918.781611] LNet: Removed LNI 192.168.203.144@tcp18 [14918.790311] LNet: Removed LNI 192.168.203.144@tcp19 [14918.795845] LNet: Removed LNI 192.168.203.144@tcp20 [14918.802383] LNet: Removed LNI 192.168.203.144@tcp21 [14918.808822] LNet: Removed LNI 192.168.203.144@tcp22 [14918.819803] LNet: Removed LNI 192.168.203.144@tcp23 [14918.828472] LNet: Removed LNI 192.168.203.144@tcp24 [14918.834511] LNet: Removed LNI 192.168.203.144@tcp25 [14918.841542] LNet: Removed LNI 192.168.203.144@tcp26 [14918.846848] LNet: Removed LNI 192.168.203.144@tcp27 [14918.856284] LNet: Removed LNI 192.168.203.144@tcp28 [14918.861443] LNet: Removed LNI 192.168.203.144@tcp29 [14918.872216] LNet: Removed LNI 192.168.203.144@tcp30 [14918.874253] LNet: Removed LNI 192.168.203.144@tcp31 [14918.885358] LNet: Removed LNI 192.168.203.144@tcp32 [14918.891553] LNet: Removed LNI 192.168.203.144@tcp33 [14918.897711] LNet: Removed LNI 192.168.203.144@tcp34 [14918.904310] LNet: Removed LNI 192.168.203.144@tcp35 [14918.906061] LNet: Removed LNI 192.168.203.144@tcp36 [14918.913995] LNet: Removed LNI 192.168.203.144@tcp37 [14918.920551] LNet: Removed LNI 192.168.203.144@tcp38 [14918.926718] LNet: Removed LNI 192.168.203.144@tcp39 [14918.936362] LNet: Removed LNI 192.168.203.144@tcp40 [14918.941233] LNet: Removed LNI 192.168.203.144@tcp41 [14918.946713] LNet: Removed LNI 192.168.203.144@tcp42 [14918.952341] LNet: Removed LNI 192.168.203.144@tcp43 [14918.955869] LNet: Removed LNI 192.168.203.144@tcp44 [14918.969739] LNet: Removed LNI 192.168.203.144@tcp45 [14918.981136] LNet: Removed LNI 192.168.203.144@tcp46 [14918.983225] LNet: Removed LNI 192.168.203.144@tcp47 [14918.991323] LNet: Removed LNI 192.168.203.144@tcp48 [14918.999993] LNet: Removed LNI 192.168.203.144@tcp49 [14919.016599] LNet: Removed LNI 192.168.203.144@tcp50 [14920.151134] Key type .llcrypt unregistered [14920.156464] Key type ._llcrypt unregistered [14947.798322] Key type ._llcrypt registered [14947.799779] Key type .llcrypt registered [14947.869670] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [14965.343882] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [14967.658296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14967.677831] alg: No test for adler32 (adler32-zlib) [14968.788272] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [14969.141519] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [14970.784067] Key type lgssc registered [14971.898075] Lustre: Echo OBD driver; http://www.lustre.org/ [15028.013653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15028.106878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15029.793749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15029.904949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15030.035499] Lustre: lustre-MDT0000: new disk, initializing [15030.303677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15030.346165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15036.618119] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15047.942832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15048.054322] Lustre: 505689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15048.139935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15048.143428] Lustre: Skipped 1 previous similar message [15048.198346] Lustre: lustre-MDT0001: new disk, initializing [15048.267942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15048.319553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15048.332376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15052.812731] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15063.732111] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15072.546980] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15084.452988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15084.850149] Lustre: *** cfs_fail_loc=90f, val=0*** [15085.011385] Lustre: lustre-OST0000: new disk, initializing [15085.018544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15085.031340] Lustre: 507709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15085.197341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15090.329377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15090.345449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15090.574524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15093.236635] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15103.920529] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15116.258209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15116.264619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15116.274834] LustreError: Skipped 1 previous similar message [15116.291557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15116.315500] Lustre: Skipped 1 previous similar message [15119.522973] Lustre: server umount lustre-OST0000 complete [15124.969772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15124.975769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15124.995403] Lustre: Skipped 1 previous similar message [15125.003397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15126.496967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15126.498692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15126.509520] Lustre: Skipped 1 previous similar message [15129.737639] Lustre: server umount lustre-MDT0000 complete [15133.286511] LustreError: 505680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877240 with bad export cookie 17571935031658171858 [15133.288086] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15133.309529] LustreError: 505680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15133.774513] Lustre: server umount lustre-MDT0001 complete [15152.443374] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15156.290149] Key type lgssc unregistered [15156.544983] LNet: 509623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15156.562287] LNetError: 509623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15156.581513] LNet: Removed LNI 192.168.203.144@tcp [15157.478745] Key type .llcrypt unregistered [15157.482645] Key type ._llcrypt unregistered [15180.358509] Key type ._llcrypt registered [15180.367068] Key type .llcrypt registered [15180.594125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15181.841686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15181.995333] alg: No test for adler32 (adler32-zlib) [15183.135505] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [15183.407211] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [15185.111382] Key type lgssc registered [15186.615207] Lustre: Echo OBD driver; http://www.lustre.org/ [15195.181608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15195.210951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15196.616783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15201.020688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15206.893925] LustreError: 510900:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15206.933983] LustreError: 510900:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15212.016157] LustreError: 510901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15212.526472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15213.180182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15218.958251] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15227.506720] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15234.459470] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15242.031774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15242.162297] LustreError: 512721:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15242.443494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15250.174592] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15258.622848] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15273.854724] loop0: detected capacity change from 0 to 204800000 [15274.077396] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15274.101338] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15274.133809] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15281.524060] loop0: detected capacity change from 0 to 204800000 [15281.696187] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15281.705928] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15281.724925] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15287.187199] loop0: detected capacity change from 0 to 204800000 [15293.350505] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15293.472324] Lustre: fs15246-OST0000: new disk, initializing [15293.476733] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15293.482357] Lustre: 514496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15293.562786] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15303.296230] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15318.989461] loop1: detected capacity change from 0 to 204800000 [15326.485928] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15326.639261] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15326.692924] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15326.785980] Lustre: fs15246-MDT0000: new disk, initializing [15327.019970] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15327.056263] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15327.248179] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15328.288764] Lustre: 515619:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004c6e5fc5 x1874725217291648/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787877451 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15330.871720] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15333.361078] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15333.379986] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15333.454096] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15352.814326] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15352.845538] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15357.937686] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15358.851896] Lustre: server umount fs15246-MDT0000 complete [15363.064675] LustreError: 510896:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15368.168689] LustreError: 510896:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15368.529728] Lustre: server umount fs15246-OST0000 complete [15380.961289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15380.969930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15380.971558] LustreError: Skipped 1 previous similar message [15380.973755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15380.999533] Lustre: Skipped 1 previous similar message [15383.522326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15383.534486] Lustre: Skipped 1 previous similar message [15385.424660] Lustre: server umount lustre-OST0000 complete [15392.238687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15392.252480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15392.288633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15397.065739] Lustre: server umount lustre-MDT0000 complete [15398.880418] LustreError: 510901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15398.912585] LustreError: 510901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [15400.923294] LustreError: 510881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877507 with bad export cookie 231236212901665031 [15400.924981] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15400.941560] LustreError: 510881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15401.308910] Lustre: server umount lustre-MDT0001 complete [15413.927681] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15416.733445] Key type lgssc unregistered [15417.028268] LNet: 518722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15417.036511] LNetError: 518722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15417.051315] LNet: Removed LNI 192.168.203.144@tcp [15418.130891] Key type .llcrypt unregistered [15418.134306] Key type ._llcrypt unregistered [15441.251829] Key type ._llcrypt registered [15441.255292] Key type .llcrypt registered [15441.437321] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15442.720960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15442.874316] alg: No test for adler32 (adler32-zlib) [15444.062434] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [15444.353957] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [15445.988035] Key type lgssc registered [15447.330784] Lustre: Echo OBD driver; http://www.lustre.org/ [15488.745156] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [15498.529611] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15538.762075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15538.795947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15540.212131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15540.267694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15540.425048] Lustre: lustre-MDT0000: new disk, initializing [15540.513204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15540.535336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15546.432539] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15558.051603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15558.186140] Lustre: 523210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15558.260889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15558.264764] Lustre: Skipped 1 previous similar message [15558.337596] Lustre: lustre-MDT0001: new disk, initializing [15558.501506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15558.564123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15558.585687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15564.254889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15574.355866] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15580.424877] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15587.637459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15587.863793] Lustre: lustre-OST0000: new disk, initializing [15587.872973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15587.895910] Lustre: 525174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15587.995907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15597.181756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [15597.676332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15597.697516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15597.820492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15610.249886] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15615.045133] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15615.369547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15619.890288] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15620.336268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15623.651632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15623.660585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15623.660684] Lustre: Skipped 1 previous similar message [15623.661605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15623.678885] Lustre: Skipped 1 previous similar message [15628.707769] Lustre: server umount lustre-OST0000 complete [15633.891286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.898152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15633.901033] Lustre: Skipped 1 previous similar message [15639.014350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15639.019479] Lustre: Skipped 1 previous similar message [15644.129193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15644.144459] Lustre: Skipped 1 previous similar message [15647.201238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15647.561156] Lustre: server umount lustre-MDT0000 complete [15651.673579] LustreError: 523202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877758 with bad export cookie 16186997928712677569 [15651.682211] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15651.687587] LustreError: 523202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15652.038691] Lustre: server umount lustre-MDT0001 complete [15680.218541] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [15697.452228] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [15736.100695] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15477 sec ========= 20:44:00 (1787877840) [15739.439381] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:44:02 (1787877842) === [15745.805288] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:44:09 (1787877849) === [15790.793357] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [15794.191488] Key type lgssc unregistered [15794.618549] LNet: 531849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15794.630142] LNetError: 531849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15794.678954] LNet: Removed LNI 192.168.203.144@tcp [15795.929668] Key type .llcrypt unregistered [15795.938252] Key type ._llcrypt unregistered